2021-09-02T07:16:22.105 INFO:root:teuthology version: 1.1.0-df69d6db 2021-09-02T07:16:22.107 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_soumyakoduri@teuthology --verbose --name soumyakoduri-2021-09-02_01:20:40-rgw-wip-skoduri-dbstore-object-distro-basic-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi} --archive /home/teuthworker/archive/soumyakoduri-2021-09-02_01:20:40-rgw-wip-skoduri-dbstore-object-distro-basic-smithi/6371861 -- /home/teuthworker/archive/soumyakoduri-2021-09-02_01:20:40-rgw-wip-skoduri-dbstore-object-distro-basic-smithi/6371861/orig.config.yaml 2021-09-02T07:16:22.152 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-09-02T07:16:22.216 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/soumyakoduri-2021-09-02_01:20:40-rgw-wip-skoduri-dbstore-object-distro-basic-smithi/6371861 branch: wip-skoduri-dbstore-object description: rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi} email: null first_in_suite: false job_id: '6371861' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: soumyakoduri-2021-09-02_01:20:40-rgw-wip-skoduri-dbstore-object-distro-basic-smithi nuke-on-error: true os_type: centos os_version: '8.3' overrides: admin_socket: branch: wip-skoduri-dbstore-object ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 9366f375ac31713c35e559edbd160565b784db1d wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 9366f375ac31713c35e559edbd160565b784db1d rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 rgw-multisite-tests: args: - tests.py - tests_ps.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-skoduri-dbstore-object sha1: 9366f375ac31713c35e559edbd160565b784db1d owner: scheduled_soumyakoduri@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 9366f375ac31713c35e559edbd160565b784db1d sleep_before_teardown: 0 suite: rgw suite_branch: wip-skoduri-dbstore-object suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_9366f375ac31713c35e559edbd160565b784db1d/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 9366f375ac31713c35e559edbd160565b784db1d targets: smithi175.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtuWE9inORipzeE+c+qLMP7PU0ecZk/wcl3RE/RP6rStzOx64hb5BodiNDj49HIXSUwRgPDulx5ybgPNn7GbpZtR7Cf9V5PRoeIXwuW8aP6+Zl7H4zba6Xqh7fQ0w3f3yM45Mx9rPUFZtxNL/PS91bh5HU6ZAv1AUKBl3tQePOsxOBg2LJqFLov7K6bG71BDb6shOEaq6f+ZUaJE7+4HX2msifQnRmuq0OXWrpxAbTmqdAbAS9CP3Aof9f1w+6g12XTkTrB2UhRBiDLgjLj6osOXyXWetUMJ9WHCyfDOhtbavgpkjQWwDH5PWwbGAhHMmlXFD/siT2vevCswIWGaWdKimy2A1o/cMcS9v2slsiPeoiQWBbBkvO3rouamC3hUEwCZYbNduTOdnngQLfzxxJ2fiBFyXuCouLGL3eP06Bei1opwh9GIjq3iSMCJVA4OJeNw2jp0Zc22mUq3LO/dgsYFygus1flReNTmTlq4s0mAKoHYFPN3NkMLktBo62dW8= smithi197.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5bB3TaCFQfuTSeKh7NnjyOp1hVDtEXsXv52fZPgcyTE20csjTrt9TJJJfjAYPwU4o3iukgEiMnem+NMjtU+9SJZ2PPGOY0MwzazJCuotZ54nUPf9OaJI8lu4NVq26s7PkEWXa8+pwCmZ9hMBt7Ww216VdaAUgectXUk93NVjZAC0Nd387dL1+iT3rwaRjelbeKwkHCxyOIs+k7J3oPkZ78HKR1IlSUwHok1tdNyOWa0L7cqwoGrDslBdZ3+QYXIyMZtX8LWlRimaAjyYc53aOBn81EecHmVoSG6CxWiA94ZW+x+2/ftcObopTJWSxNP1ywgY+JtDGLjxlLKJkKNtISZ8Zg4NZEm31xV3vmXeqihw+AdX+pmHKwZeC31bmkcN/2c0bVcScBrno8DsBL3C2CgC1d3xtvaw9VcUp85m/Q8m8h2NNxGVDuKcm/BtPIKqbl+ItW7629pp0YcgNka+VKwn17KWN0fFTl+ghKKXc9BE1ynWEE4VwL6xJjeU8kds= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master timestamp: 2021-09-02_01:20:40 tube: smithi user: soumyakoduri verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.30581 2021-09-02T07:16:22.217 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_9366f375ac31713c35e559edbd160565b784db1d/qa; will attempt to use it 2021-09-02T07:16:22.218 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_9366f375ac31713c35e559edbd160565b784db1d/qa/tasks 2021-09-02T07:16:22.218 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-09-02T07:16:22.220 INFO:teuthology.task.internal:Checking packages... 2021-09-02T07:16:22.241 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '9366f375ac31713c35e559edbd160565b784db1d' 2021-09-02T07:16:22.242 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-09-02T07:16:22.242 INFO:teuthology.packaging:ref: None 2021-09-02T07:16:22.242 INFO:teuthology.packaging:tag: None 2021-09-02T07:16:22.243 INFO:teuthology.packaging:branch: wip-skoduri-dbstore-object 2021-09-02T07:16:22.243 INFO:teuthology.packaging:sha1: 9366f375ac31713c35e559edbd160565b784db1d 2021-09-02T07:16:22.244 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-skoduri-dbstore-object 2021-09-02T07:16:22.777 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-7401.g9366f375 2021-09-02T07:16:22.778 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-09-02T07:16:22.788 INFO:teuthology.task.internal:no buildpackages task found 2021-09-02T07:16:22.788 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-09-02T07:16:22.827 INFO:teuthology.task.internal:Saving configuration 2021-09-02T07:16:22.859 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-09-02T07:16:22.899 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-02T07:16:22.917 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-09-02 07:11:28.170071', 'description': '/home/teuthworker/archive/soumyakoduri-2021-09-02_01:20:40-rgw-wip-skoduri-dbstore-object-distro-basic-smithi/6371861', 'locked_by': 'scheduled_soumyakoduri@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMOJbyC8SnCoKn1jrLAERgtqU02cXuTjkBB+SaHp4ywYX/ry0jZthhTHW+ZUqotxDJTsJZ1/6fdq7D+IiL5t/xA=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi175.front.sepia.ceph.com'} 2021-09-02T07:16:22.934 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-09-02 07:11:28.170050', 'description': '/home/teuthworker/archive/soumyakoduri-2021-09-02_01:20:40-rgw-wip-skoduri-dbstore-object-distro-basic-smithi/6371861', 'locked_by': 'scheduled_soumyakoduri@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAOoiUGxyHqYxup3qDIqfOdoccRnKHnjZiwpZr7dD4+Dn3kPMUEySvnzxxg1AuEMKpgLKNjnRhtvHep3wchXwGU=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi197.front.sepia.ceph.com'} 2021-09-02T07:16:22.934 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-09-02T07:16:22.944 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2021-09-02T07:16:22.944 INFO:teuthology.task.internal:roles: ubuntu@smithi197.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2021-09-02T07:16:22.944 INFO:teuthology.run_tasks:Running task console_log... 2021-09-02T07:16:23.027 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd2f4d1dd90>, signals=[15]) 2021-09-02T07:16:23.028 INFO:teuthology.run_tasks:Running task internal.connect... 2021-09-02T07:16:23.037 INFO:teuthology.task.internal:Opening connections... 2021-09-02T07:16:23.038 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi175.front.sepia.ceph.com 2021-09-02T07:16:23.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-02T07:16:23.109 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi197.front.sepia.ceph.com 2021-09-02T07:16:23.110 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi197.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-02T07:16:23.178 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-09-02T07:16:23.191 DEBUG:teuthology.orchestra.run.smithi175:> uname -m 2021-09-02T07:16:23.214 INFO:teuthology.orchestra.run.smithi175.stdout:x86_64 2021-09-02T07:16:23.215 DEBUG:teuthology.orchestra.run.smithi175:> cat /etc/os-release 2021-09-02T07:16:23.269 INFO:teuthology.orchestra.run.smithi175.stdout:NAME="CentOS Linux" 2021-09-02T07:16:23.270 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION="8" 2021-09-02T07:16:23.270 INFO:teuthology.orchestra.run.smithi175.stdout:ID="centos" 2021-09-02T07:16:23.270 INFO:teuthology.orchestra.run.smithi175.stdout:ID_LIKE="rhel fedora" 2021-09-02T07:16:23.270 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION_ID="8" 2021-09-02T07:16:23.271 INFO:teuthology.orchestra.run.smithi175.stdout:PLATFORM_ID="platform:el8" 2021-09-02T07:16:23.271 INFO:teuthology.orchestra.run.smithi175.stdout:PRETTY_NAME="CentOS Linux 8" 2021-09-02T07:16:23.271 INFO:teuthology.orchestra.run.smithi175.stdout:ANSI_COLOR="0;31" 2021-09-02T07:16:23.271 INFO:teuthology.orchestra.run.smithi175.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-09-02T07:16:23.271 INFO:teuthology.orchestra.run.smithi175.stdout:HOME_URL="https://centos.org/" 2021-09-02T07:16:23.272 INFO:teuthology.orchestra.run.smithi175.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-09-02T07:16:23.272 INFO:teuthology.orchestra.run.smithi175.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-09-02T07:16:23.272 INFO:teuthology.orchestra.run.smithi175.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-09-02T07:16:23.273 INFO:teuthology.lock.ops:Updating smithi175.front.sepia.ceph.com on lock server 2021-09-02T07:16:23.291 DEBUG:teuthology.orchestra.run.smithi197:> uname -m 2021-09-02T07:16:23.307 INFO:teuthology.orchestra.run.smithi197.stdout:x86_64 2021-09-02T07:16:23.308 DEBUG:teuthology.orchestra.run.smithi197:> cat /etc/os-release 2021-09-02T07:16:23.362 INFO:teuthology.orchestra.run.smithi197.stdout:NAME="CentOS Linux" 2021-09-02T07:16:23.363 INFO:teuthology.orchestra.run.smithi197.stdout:VERSION="8" 2021-09-02T07:16:23.363 INFO:teuthology.orchestra.run.smithi197.stdout:ID="centos" 2021-09-02T07:16:23.364 INFO:teuthology.orchestra.run.smithi197.stdout:ID_LIKE="rhel fedora" 2021-09-02T07:16:23.364 INFO:teuthology.orchestra.run.smithi197.stdout:VERSION_ID="8" 2021-09-02T07:16:23.364 INFO:teuthology.orchestra.run.smithi197.stdout:PLATFORM_ID="platform:el8" 2021-09-02T07:16:23.364 INFO:teuthology.orchestra.run.smithi197.stdout:PRETTY_NAME="CentOS Linux 8" 2021-09-02T07:16:23.364 INFO:teuthology.orchestra.run.smithi197.stdout:ANSI_COLOR="0;31" 2021-09-02T07:16:23.365 INFO:teuthology.orchestra.run.smithi197.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-09-02T07:16:23.365 INFO:teuthology.orchestra.run.smithi197.stdout:HOME_URL="https://centos.org/" 2021-09-02T07:16:23.365 INFO:teuthology.orchestra.run.smithi197.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-09-02T07:16:23.365 INFO:teuthology.orchestra.run.smithi197.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-09-02T07:16:23.366 INFO:teuthology.orchestra.run.smithi197.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-09-02T07:16:23.366 INFO:teuthology.lock.ops:Updating smithi197.front.sepia.ceph.com on lock server 2021-09-02T07:16:23.382 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-09-02T07:16:23.395 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-09-02T07:16:23.418 INFO:teuthology.task.internal:Checking for old test directory... 2021-09-02T07:16:23.419 DEBUG:teuthology.orchestra.run.smithi175:> test '!' -e /home/ubuntu/cephtest 2021-09-02T07:16:23.422 DEBUG:teuthology.orchestra.run.smithi197:> test '!' -e /home/ubuntu/cephtest 2021-09-02T07:16:23.436 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-09-02T07:16:23.453 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-09-02T07:16:23.454 DEBUG:teuthology.orchestra.run.smithi175:> test -z $(ls -A /var/lib/ceph) 2021-09-02T07:16:23.476 DEBUG:teuthology.orchestra.run.smithi197:> test -z $(ls -A /var/lib/ceph) 2021-09-02T07:16:23.504 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-09-02T07:16:23.555 INFO:teuthology.run_tasks:Running task kernel... 2021-09-02T07:16:23.576 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2021-09-02T07:16:23.576 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-09-02T07:16:23.577 DEBUG:teuthology.orchestra.run.smithi175:> uname -r 2021-09-02T07:16:23.591 INFO:teuthology.orchestra.run.smithi175.stdout:4.18.0-305.10.2.el8_4.x86_64 2021-09-02T07:16:23.591 INFO:teuthology.task.kernel:Running kernel on smithi175: 4.18.0-305.10.2.el8_4.x86_64 2021-09-02T07:16:23.592 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum install -y kernel 2021-09-02T07:16:25.866 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS-8 - AppStream 39 MB/s | 8.8 MB 00:00 2021-09-02T07:16:27.520 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS-8 - Base 19 MB/s | 5.6 MB 00:00 2021-09-02T07:16:28.364 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS-8 - Extras 164 kB/s | 10 kB 00:00 2021-09-02T07:16:32.233 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Linux 8 - AppStream 2.3 MB/s | 8.8 MB 00:03 2021-09-02T07:16:34.261 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Linux 8 - BaseOS 8.3 MB/s | 5.6 MB 00:00 2021-09-02T07:16:35.308 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Linux 8 - Extras 37 kB/s | 10 kB 00:00 2021-09-02T07:16:35.454 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS-8 - PowerTools 25 MB/s | 2.3 MB 00:00 2021-09-02T07:16:38.133 INFO:teuthology.orchestra.run.smithi175.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2021-09-02T07:16:39.102 INFO:teuthology.orchestra.run.smithi175.stdout:Extra Packages for Enterprise Linux 11 MB/s | 10 MB 00:00 2021-09-02T07:16:41.267 INFO:teuthology.orchestra.run.smithi175.stdout:lab-extras 4.7 MB/s | 24 kB 00:00 2021-09-02T07:16:43.155 INFO:teuthology.orchestra.run.smithi175.stdout:Package kernel-4.18.0-305.10.2.el8_4.x86_64 is already installed. 2021-09-02T07:16:43.156 INFO:teuthology.orchestra.run.smithi175.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-09-02T07:16:43.452 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:16:43.452 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:16:43.452 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2021-09-02T07:16:43.452 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:16:43.453 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2021-09-02T07:16:43.453 INFO:teuthology.orchestra.run.smithi175.stdout: kernel x86_64 4.18.0-305.12.1.el8_4 CentOS-Base 5.9 M 2021-09-02T07:16:43.453 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2021-09-02T07:16:43.453 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-core x86_64 4.18.0-305.12.1.el8_4 CentOS-Base 36 M 2021-09-02T07:16:43.454 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-modules x86_64 4.18.0-305.12.1.el8_4 CentOS-Base 28 M 2021-09-02T07:16:43.454 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:16:43.454 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2021-09-02T07:16:43.454 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:16:43.454 INFO:teuthology.orchestra.run.smithi175.stdout:Install 3 Packages 2021-09-02T07:16:43.455 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:16:43.455 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 70 M 2021-09-02T07:16:43.455 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 88 M 2021-09-02T07:16:43.455 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2021-09-02T07:16:43.875 INFO:teuthology.orchestra.run.smithi175.stdout:(1/3): kernel-4.18.0-305.12.1.el8_4.x86_64.rpm 14 MB/s | 5.9 MB 00:00 2021-09-02T07:16:44.205 INFO:teuthology.orchestra.run.smithi175.stdout:(2/3): kernel-core-4.18.0-305.12.1.el8_4.x86_64 48 MB/s | 36 MB 00:00 2021-09-02T07:16:44.745 INFO:teuthology.orchestra.run.smithi175.stdout:(3/3): kernel-modules-4.18.0-305.12.1.el8_4.x86 22 MB/s | 28 MB 00:01 2021-09-02T07:16:44.749 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:16:44.749 INFO:teuthology.orchestra.run.smithi175.stdout:Total 54 MB/s | 70 MB 00:01 2021-09-02T07:16:44.862 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2021-09-02T07:16:45.403 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2021-09-02T07:16:45.403 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2021-09-02T07:16:46.568 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2021-09-02T07:16:46.587 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2021-09-02T07:16:49.011 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2021-09-02T07:16:49.511 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : kernel-core-4.18.0-305.12.1.el8_4.x86_64 1/3 2021-09-02T07:16:52.732 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-core-4.18.0-305.12.1.el8_4.x86_64 1/3 2021-09-02T07:16:53.085 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : kernel-modules-4.18.0-305.12.1.el8_4.x86_64 2/3 2021-09-02T07:16:56.053 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-modules-4.18.0-305.12.1.el8_4.x86_64 2/3 2021-09-02T07:16:56.255 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : kernel-4.18.0-305.12.1.el8_4.x86_64 3/3 2021-09-02T07:17:16.622 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-core-4.18.0-305.12.1.el8_4.x86_64 3/3 2021-09-02T07:17:16.924 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-4.18.0-305.12.1.el8_4.x86_64 3/3 2021-09-02T07:17:16.924 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : kernel-4.18.0-305.12.1.el8_4.x86_64 1/3 2021-09-02T07:17:16.924 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : kernel-core-4.18.0-305.12.1.el8_4.x86_64 2/3 2021-09-02T07:17:17.331 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : kernel-modules-4.18.0-305.12.1.el8_4.x86_64 3/3 2021-09-02T07:17:17.332 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:17:17.332 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2021-09-02T07:17:17.332 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-4.18.0-305.12.1.el8_4.x86_64 2021-09-02T07:17:17.332 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-core-4.18.0-305.12.1.el8_4.x86_64 2021-09-02T07:17:17.333 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-modules-4.18.0-305.12.1.el8_4.x86_64 2021-09-02T07:17:17.333 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:17:17.333 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:17:17.397 DEBUG:teuthology.orchestra.run.smithi175:> rpm -q kernel --last | head -n 1 2021-09-02T07:17:17.539 INFO:teuthology.orchestra.run.smithi175.stdout:kernel-4.18.0-305.12.1.el8_4.x86_64 Thu 02 Sep 2021 07:16:56 AM UTC 2021-09-02T07:17:17.541 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.12.1.el8_4.x86_64 2021-09-02T07:17:17.541 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.10.2.el8_4.x86_64 Expected: 4.18.0-305.12.1.el8_4.x86_64 2021-09-02T07:17:17.542 DEBUG:teuthology.orchestra.run.smithi197:> uname -r 2021-09-02T07:17:17.561 INFO:teuthology.orchestra.run.smithi197.stdout:4.18.0-305.10.2.el8_4.x86_64 2021-09-02T07:17:17.561 INFO:teuthology.task.kernel:Running kernel on smithi197: 4.18.0-305.10.2.el8_4.x86_64 2021-09-02T07:17:17.561 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum install -y kernel 2021-09-02T07:17:19.828 INFO:teuthology.orchestra.run.smithi197.stdout:CentOS-8 - AppStream 45 MB/s | 8.8 MB 00:00 2021-09-02T07:17:21.369 INFO:teuthology.orchestra.run.smithi197.stdout:CentOS-8 - Base 44 MB/s | 5.6 MB 00:00 2021-09-02T07:17:22.210 INFO:teuthology.orchestra.run.smithi197.stdout:CentOS-8 - Extras 162 kB/s | 10 kB 00:00 2021-09-02T07:17:26.088 INFO:teuthology.orchestra.run.smithi197.stdout:CentOS Linux 8 - AppStream 2.3 MB/s | 8.8 MB 00:03 2021-09-02T07:17:28.201 INFO:teuthology.orchestra.run.smithi197.stdout:CentOS Linux 8 - BaseOS 8.8 MB/s | 5.6 MB 00:00 2021-09-02T07:17:29.149 INFO:teuthology.orchestra.run.smithi197.stdout:CentOS Linux 8 - Extras 67 kB/s | 10 kB 00:00 2021-09-02T07:17:29.293 INFO:teuthology.orchestra.run.smithi197.stdout:CentOS-8 - PowerTools 26 MB/s | 2.3 MB 00:00 2021-09-02T07:17:31.990 INFO:teuthology.orchestra.run.smithi197.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2021-09-02T07:17:32.910 INFO:teuthology.orchestra.run.smithi197.stdout:Extra Packages for Enterprise Linux 12 MB/s | 10 MB 00:00 2021-09-02T07:17:35.092 INFO:teuthology.orchestra.run.smithi197.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2021-09-02T07:17:37.068 INFO:teuthology.orchestra.run.smithi197.stdout:Package kernel-4.18.0-305.10.2.el8_4.x86_64 is already installed. 2021-09-02T07:17:37.068 INFO:teuthology.orchestra.run.smithi197.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-09-02T07:17:37.379 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:17:37.380 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:17:37.380 INFO:teuthology.orchestra.run.smithi197.stdout: Package Arch Version Repository Size 2021-09-02T07:17:37.380 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:17:37.380 INFO:teuthology.orchestra.run.smithi197.stdout:Installing: 2021-09-02T07:17:37.381 INFO:teuthology.orchestra.run.smithi197.stdout: kernel x86_64 4.18.0-305.12.1.el8_4 CentOS-Base 5.9 M 2021-09-02T07:17:37.381 INFO:teuthology.orchestra.run.smithi197.stdout:Installing dependencies: 2021-09-02T07:17:37.381 INFO:teuthology.orchestra.run.smithi197.stdout: kernel-core x86_64 4.18.0-305.12.1.el8_4 CentOS-Base 36 M 2021-09-02T07:17:37.381 INFO:teuthology.orchestra.run.smithi197.stdout: kernel-modules x86_64 4.18.0-305.12.1.el8_4 CentOS-Base 28 M 2021-09-02T07:17:37.382 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:17:37.382 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction Summary 2021-09-02T07:17:37.382 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:17:37.382 INFO:teuthology.orchestra.run.smithi197.stdout:Install 3 Packages 2021-09-02T07:17:37.382 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:17:37.383 INFO:teuthology.orchestra.run.smithi197.stdout:Total download size: 70 M 2021-09-02T07:17:37.383 INFO:teuthology.orchestra.run.smithi197.stdout:Installed size: 88 M 2021-09-02T07:17:37.384 INFO:teuthology.orchestra.run.smithi197.stdout:Downloading Packages: 2021-09-02T07:17:37.554 INFO:teuthology.orchestra.run.smithi197.stdout:(1/3): kernel-4.18.0-305.12.1.el8_4.x86_64.rpm 34 MB/s | 5.9 MB 00:00 2021-09-02T07:17:38.208 INFO:teuthology.orchestra.run.smithi197.stdout:(2/3): kernel-core-4.18.0-305.12.1.el8_4.x86_64 43 MB/s | 36 MB 00:00 2021-09-02T07:17:38.524 INFO:teuthology.orchestra.run.smithi197.stdout:(3/3): kernel-modules-4.18.0-305.12.1.el8_4.x86 25 MB/s | 28 MB 00:01 2021-09-02T07:17:38.528 INFO:teuthology.orchestra.run.smithi197.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:17:38.528 INFO:teuthology.orchestra.run.smithi197.stdout:Total 61 MB/s | 70 MB 00:01 2021-09-02T07:17:38.640 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction check 2021-09-02T07:17:38.998 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction check succeeded. 2021-09-02T07:17:38.998 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction test 2021-09-02T07:17:40.188 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction test succeeded. 2021-09-02T07:17:40.222 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction 2021-09-02T07:17:42.660 INFO:teuthology.orchestra.run.smithi197.stdout: Preparing : 1/1 2021-09-02T07:17:43.248 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : kernel-core-4.18.0-305.12.1.el8_4.x86_64 1/3 2021-09-02T07:17:46.520 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: kernel-core-4.18.0-305.12.1.el8_4.x86_64 1/3 2021-09-02T07:17:46.880 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : kernel-modules-4.18.0-305.12.1.el8_4.x86_64 2/3 2021-09-02T07:17:49.848 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: kernel-modules-4.18.0-305.12.1.el8_4.x86_64 2/3 2021-09-02T07:17:50.075 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : kernel-4.18.0-305.12.1.el8_4.x86_64 3/3 2021-09-02T07:18:10.545 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: kernel-core-4.18.0-305.12.1.el8_4.x86_64 3/3 2021-09-02T07:18:10.832 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: kernel-4.18.0-305.12.1.el8_4.x86_64 3/3 2021-09-02T07:18:10.833 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : kernel-4.18.0-305.12.1.el8_4.x86_64 1/3 2021-09-02T07:18:10.833 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : kernel-core-4.18.0-305.12.1.el8_4.x86_64 2/3 2021-09-02T07:18:11.269 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : kernel-modules-4.18.0-305.12.1.el8_4.x86_64 3/3 2021-09-02T07:18:11.269 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:18:11.269 INFO:teuthology.orchestra.run.smithi197.stdout:Installed: 2021-09-02T07:18:11.270 INFO:teuthology.orchestra.run.smithi197.stdout: kernel-4.18.0-305.12.1.el8_4.x86_64 2021-09-02T07:18:11.270 INFO:teuthology.orchestra.run.smithi197.stdout: kernel-core-4.18.0-305.12.1.el8_4.x86_64 2021-09-02T07:18:11.270 INFO:teuthology.orchestra.run.smithi197.stdout: kernel-modules-4.18.0-305.12.1.el8_4.x86_64 2021-09-02T07:18:11.270 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:18:11.271 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:18:11.329 DEBUG:teuthology.orchestra.run.smithi197:> rpm -q kernel --last | head -n 1 2021-09-02T07:18:11.480 INFO:teuthology.orchestra.run.smithi197.stdout:kernel-4.18.0-305.12.1.el8_4.x86_64 Thu 02 Sep 2021 07:17:49 AM UTC 2021-09-02T07:18:11.481 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.12.1.el8_4.x86_64 2021-09-02T07:18:11.481 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.10.2.el8_4.x86_64 Expected: 4.18.0-305.12.1.el8_4.x86_64 2021-09-02T07:18:11.482 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-09-02T07:18:11.482 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-09-02T07:18:11.482 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-09-02T07:18:11.483 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2021-09-02T07:18:11.483 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi175.front.sepia.ceph.com, path=None, version=distro) 2021-09-02T07:18:11.483 DEBUG:teuthology.orchestra.run.smithi175:> rpm -q kernel --last | head -n 1 2021-09-02T07:18:11.622 INFO:teuthology.orchestra.run.smithi175.stdout:kernel-4.18.0-305.12.1.el8_4.x86_64 Thu 02 Sep 2021 07:16:56 AM UTC 2021-09-02T07:18:11.624 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.12.1.el8_4.x86_64 2021-09-02T07:18:11.625 DEBUG:teuthology.orchestra.run.smithi175:> sudo rpm -qi grub2-tools 2021-09-02T07:18:11.655 INFO:teuthology.orchestra.run.smithi175.stdout:Name : grub2-tools 2021-09-02T07:18:11.655 INFO:teuthology.orchestra.run.smithi175.stdout:Epoch : 1 2021-09-02T07:18:11.656 INFO:teuthology.orchestra.run.smithi175.stdout:Version : 2.02 2021-09-02T07:18:11.656 INFO:teuthology.orchestra.run.smithi175.stdout:Release : 90.el8 2021-09-02T07:18:11.656 INFO:teuthology.orchestra.run.smithi175.stdout:Architecture: x86_64 2021-09-02T07:18:11.656 INFO:teuthology.orchestra.run.smithi175.stdout:Install Date: Wed 11 Aug 2021 06:38:04 PM UTC 2021-09-02T07:18:11.657 INFO:teuthology.orchestra.run.smithi175.stdout:Group : System Environment/Base 2021-09-02T07:18:11.657 INFO:teuthology.orchestra.run.smithi175.stdout:Size : 9197207 2021-09-02T07:18:11.658 INFO:teuthology.orchestra.run.smithi175.stdout:License : GPLv3+ 2021-09-02T07:18:11.658 INFO:teuthology.orchestra.run.smithi175.stdout:Signature : RSA/SHA256, Tue 08 Sep 2020 07:14:39 PM UTC, Key ID 05b555b38483c65d 2021-09-02T07:18:11.658 INFO:teuthology.orchestra.run.smithi175.stdout:Source RPM : grub2-2.02-90.el8.src.rpm 2021-09-02T07:18:11.658 INFO:teuthology.orchestra.run.smithi175.stdout:Build Date : Tue 08 Sep 2020 07:00:29 PM UTC 2021-09-02T07:18:11.659 INFO:teuthology.orchestra.run.smithi175.stdout:Build Host : kbuilder.bsys.centos.org 2021-09-02T07:18:11.659 INFO:teuthology.orchestra.run.smithi175.stdout:Relocations : (not relocatable) 2021-09-02T07:18:11.659 INFO:teuthology.orchestra.run.smithi175.stdout:Packager : CentOS BuildSystem 2021-09-02T07:18:11.660 INFO:teuthology.orchestra.run.smithi175.stdout:Vendor : CentOS 2021-09-02T07:18:11.660 INFO:teuthology.orchestra.run.smithi175.stdout:URL : http://www.gnu.org/software/grub/ 2021-09-02T07:18:11.660 INFO:teuthology.orchestra.run.smithi175.stdout:Summary : Support tools for GRUB. 2021-09-02T07:18:11.660 INFO:teuthology.orchestra.run.smithi175.stdout:Description : 2021-09-02T07:18:11.661 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:18:11.661 INFO:teuthology.orchestra.run.smithi175.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-09-02T07:18:11.661 INFO:teuthology.orchestra.run.smithi175.stdout:customizable bootloader with modular architecture. It supports a rich 2021-09-02T07:18:11.661 INFO:teuthology.orchestra.run.smithi175.stdout:variety of kernel formats, file systems, computer architectures and 2021-09-02T07:18:11.662 INFO:teuthology.orchestra.run.smithi175.stdout:hardware devices. 2021-09-02T07:18:11.662 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:18:11.662 INFO:teuthology.orchestra.run.smithi175.stdout:This subpackage provides tools for support of all platforms. 2021-09-02T07:18:11.663 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-09-02T07:18:11.663 INFO:teuthology.task.kernel:Updating grub on smithi175 to boot 4.18.0-305.12.1.el8_4.x86_64 2021-09-02T07:18:11.664 DEBUG:teuthology.orchestra.run.smithi175:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-09-02T07:18:12.406 INFO:teuthology.orchestra.run.smithi175.stderr:Generating grub configuration file ... 2021-09-02T07:18:14.862 INFO:teuthology.orchestra.run.smithi175.stderr:done 2021-09-02T07:18:14.864 DEBUG:teuthology.orchestra.run.smithi175:> mktemp 2021-09-02T07:18:14.902 INFO:teuthology.orchestra.run.smithi175.stdout:/tmp/tmp.qywfLpHnEx 2021-09-02T07:18:14.902 DEBUG:teuthology.orchestra.run.smithi175:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.qywfLpHnEx 2021-09-02T07:18:14.984 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0666 /tmp/tmp.qywfLpHnEx 2021-09-02T07:18:15.141 DEBUG:teuthology.orchestra.remote:smithi175:/tmp/tmp.qywfLpHnEx is 6KB 2021-09-02T07:18:15.191 DEBUG:teuthology.orchestra.run.smithi175:> rm -fr /tmp/tmp.qywfLpHnEx 2021-09-02T07:18:15.206 DEBUG:teuthology.orchestra.run.smithi175:> sudo /bin/ls /boot/loader/entries || true 2021-09-02T07:18:15.286 INFO:teuthology.orchestra.run.smithi175.stdout:ecc66dffcd264e5181894e69c53ff13c-0-rescue.conf 2021-09-02T07:18:15.287 INFO:teuthology.orchestra.run.smithi175.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-240.el8.x86_64.conf 2021-09-02T07:18:15.287 INFO:teuthology.orchestra.run.smithi175.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-305.10.2.el8_4.x86_64.conf 2021-09-02T07:18:15.287 INFO:teuthology.orchestra.run.smithi175.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-305.12.1.el8_4.x86_64.conf 2021-09-02T07:18:15.288 DEBUG:teuthology.orchestra.run.smithi175:> sudo grub2-set-default ecc66dffcd264e5181894e69c53ff13c-4.18.0-305.12.1.el8_4.x86_64 2021-09-02T07:18:15.432 DEBUG:teuthology.orchestra.run.smithi175:> sudo shutdown -r now 2021-09-02T07:18:15.434 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2021-09-02T07:18:15.435 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi197.front.sepia.ceph.com, path=None, version=distro) 2021-09-02T07:18:15.435 DEBUG:teuthology.orchestra.run.smithi197:> rpm -q kernel --last | head -n 1 2021-09-02T07:18:15.574 INFO:teuthology.orchestra.run.smithi197.stdout:kernel-4.18.0-305.12.1.el8_4.x86_64 Thu 02 Sep 2021 07:17:49 AM UTC 2021-09-02T07:18:15.576 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.12.1.el8_4.x86_64 2021-09-02T07:18:15.576 DEBUG:teuthology.orchestra.run.smithi197:> sudo rpm -qi grub2-tools 2021-09-02T07:18:15.606 INFO:teuthology.orchestra.run.smithi197.stdout:Name : grub2-tools 2021-09-02T07:18:15.606 INFO:teuthology.orchestra.run.smithi197.stdout:Epoch : 1 2021-09-02T07:18:15.607 INFO:teuthology.orchestra.run.smithi197.stdout:Version : 2.02 2021-09-02T07:18:15.607 INFO:teuthology.orchestra.run.smithi197.stdout:Release : 90.el8 2021-09-02T07:18:15.607 INFO:teuthology.orchestra.run.smithi197.stdout:Architecture: x86_64 2021-09-02T07:18:15.607 INFO:teuthology.orchestra.run.smithi197.stdout:Install Date: Wed 11 Aug 2021 06:38:04 PM UTC 2021-09-02T07:18:15.607 INFO:teuthology.orchestra.run.smithi197.stdout:Group : System Environment/Base 2021-09-02T07:18:15.608 INFO:teuthology.orchestra.run.smithi197.stdout:Size : 9197207 2021-09-02T07:18:15.608 INFO:teuthology.orchestra.run.smithi197.stdout:License : GPLv3+ 2021-09-02T07:18:15.608 INFO:teuthology.orchestra.run.smithi197.stdout:Signature : RSA/SHA256, Tue 08 Sep 2020 07:14:39 PM UTC, Key ID 05b555b38483c65d 2021-09-02T07:18:15.608 INFO:teuthology.orchestra.run.smithi197.stdout:Source RPM : grub2-2.02-90.el8.src.rpm 2021-09-02T07:18:15.609 INFO:teuthology.orchestra.run.smithi197.stdout:Build Date : Tue 08 Sep 2020 07:00:29 PM UTC 2021-09-02T07:18:15.609 INFO:teuthology.orchestra.run.smithi197.stdout:Build Host : kbuilder.bsys.centos.org 2021-09-02T07:18:15.609 INFO:teuthology.orchestra.run.smithi197.stdout:Relocations : (not relocatable) 2021-09-02T07:18:15.609 INFO:teuthology.orchestra.run.smithi197.stdout:Packager : CentOS BuildSystem 2021-09-02T07:18:15.609 INFO:teuthology.orchestra.run.smithi197.stdout:Vendor : CentOS 2021-09-02T07:18:15.610 INFO:teuthology.orchestra.run.smithi197.stdout:URL : http://www.gnu.org/software/grub/ 2021-09-02T07:18:15.610 INFO:teuthology.orchestra.run.smithi197.stdout:Summary : Support tools for GRUB. 2021-09-02T07:18:15.610 INFO:teuthology.orchestra.run.smithi197.stdout:Description : 2021-09-02T07:18:15.610 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:18:15.610 INFO:teuthology.orchestra.run.smithi197.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-09-02T07:18:15.611 INFO:teuthology.orchestra.run.smithi197.stdout:customizable bootloader with modular architecture. It supports a rich 2021-09-02T07:18:15.611 INFO:teuthology.orchestra.run.smithi197.stdout:variety of kernel formats, file systems, computer architectures and 2021-09-02T07:18:15.611 INFO:teuthology.orchestra.run.smithi197.stdout:hardware devices. 2021-09-02T07:18:15.611 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:18:15.611 INFO:teuthology.orchestra.run.smithi197.stdout:This subpackage provides tools for support of all platforms. 2021-09-02T07:18:15.612 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-09-02T07:18:15.613 INFO:teuthology.task.kernel:Updating grub on smithi197 to boot 4.18.0-305.12.1.el8_4.x86_64 2021-09-02T07:18:15.613 DEBUG:teuthology.orchestra.run.smithi197:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-09-02T07:18:17.105 INFO:teuthology.orchestra.run.smithi197.stderr:Generating grub configuration file ... 2021-09-02T07:18:19.998 INFO:teuthology.orchestra.run.smithi197.stderr:done 2021-09-02T07:18:20.000 DEBUG:teuthology.orchestra.run.smithi197:> mktemp 2021-09-02T07:18:20.046 INFO:teuthology.orchestra.run.smithi197.stdout:/tmp/tmp.nBSO7HgI86 2021-09-02T07:18:20.047 DEBUG:teuthology.orchestra.run.smithi197:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.nBSO7HgI86 2021-09-02T07:18:20.078 DEBUG:teuthology.orchestra.run.smithi197:> sudo chmod 0666 /tmp/tmp.nBSO7HgI86 2021-09-02T07:18:20.232 DEBUG:teuthology.orchestra.remote:smithi197:/tmp/tmp.nBSO7HgI86 is 6KB 2021-09-02T07:18:20.281 DEBUG:teuthology.orchestra.run.smithi197:> rm -fr /tmp/tmp.nBSO7HgI86 2021-09-02T07:18:20.296 DEBUG:teuthology.orchestra.run.smithi197:> sudo /bin/ls /boot/loader/entries || true 2021-09-02T07:18:20.364 INFO:teuthology.orchestra.run.smithi197.stdout:ecc66dffcd264e5181894e69c53ff13c-0-rescue.conf 2021-09-02T07:18:20.365 INFO:teuthology.orchestra.run.smithi197.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-240.el8.x86_64.conf 2021-09-02T07:18:20.365 INFO:teuthology.orchestra.run.smithi197.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-305.10.2.el8_4.x86_64.conf 2021-09-02T07:18:20.365 INFO:teuthology.orchestra.run.smithi197.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-305.12.1.el8_4.x86_64.conf 2021-09-02T07:18:20.366 DEBUG:teuthology.orchestra.run.smithi197:> sudo grub2-set-default ecc66dffcd264e5181894e69c53ff13c-4.18.0-305.12.1.el8_4.x86_64 2021-09-02T07:18:20.514 DEBUG:teuthology.orchestra.run.smithi197:> sudo shutdown -r now 2021-09-02T07:18:50.546 INFO:teuthology.misc:Re-opening connections... 2021-09-02T07:18:50.547 INFO:teuthology.misc:trying to connect to ubuntu@smithi175.front.sepia.ceph.com 2021-09-02T07:18:50.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-02T07:18:50.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-02T07:19:50.607 DEBUG:teuthology.orchestra.remote:timed out 2021-09-02T07:19:50.610 INFO:teuthology.misc:trying to connect to ubuntu@smithi197.front.sepia.ceph.com 2021-09-02T07:19:50.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-02T07:19:50.612 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi197.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-02T07:19:53.102 DEBUG:teuthology.orchestra.run.smithi197:> true 2021-09-02T07:19:53.453 DEBUG:teuthology.misc:waited 62.90544009208679 2021-09-02T07:19:54.454 INFO:teuthology.misc:trying to connect to ubuntu@smithi175.front.sepia.ceph.com 2021-09-02T07:19:54.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-02T07:19:54.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-02T07:19:54.798 DEBUG:teuthology.orchestra.run.smithi175:> true 2021-09-02T07:19:55.082 DEBUG:teuthology.misc:waited 64.53468728065491 2021-09-02T07:19:56.083 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2021-09-02T07:19:56.083 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.18.0-305.12.1.el8_4.x86_64"... 2021-09-02T07:19:56.084 DEBUG:teuthology.orchestra.run.smithi175:> uname -r 2021-09-02T07:19:56.099 INFO:teuthology.orchestra.run.smithi175.stdout:4.18.0-305.12.1.el8_4.x86_64 2021-09-02T07:19:56.100 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-305.12.1.el8_4.x86_64 vs 4.18.0-305.12.1.el8_4.x86_64 2021-09-02T07:19:56.100 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-09-02T07:19:56.100 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-09-02T07:19:56.101 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2021-09-02T07:19:56.101 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.18.0-305.12.1.el8_4.x86_64"... 2021-09-02T07:19:56.101 DEBUG:teuthology.orchestra.run.smithi197:> uname -r 2021-09-02T07:19:56.120 INFO:teuthology.orchestra.run.smithi197.stdout:4.18.0-305.12.1.el8_4.x86_64 2021-09-02T07:19:56.120 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-305.12.1.el8_4.x86_64 vs 4.18.0-305.12.1.el8_4.x86_64 2021-09-02T07:19:56.120 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-09-02T07:19:56.121 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-09-02T07:19:57.122 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2021-09-02T07:19:57.122 DEBUG:teuthology.orchestra.run.smithi175:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-09-02T07:19:57.209 INFO:teuthology.orchestra.run.smithi175.stdout:ttyS1 2021-09-02T07:19:57.235 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2021-09-02T07:19:57.235 DEBUG:teuthology.orchestra.run.smithi197:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-09-02T07:19:57.304 INFO:teuthology.orchestra.run.smithi197.stdout:ttyS1 2021-09-02T07:19:57.330 INFO:teuthology.run_tasks:Running task internal.base... 2021-09-02T07:19:57.340 INFO:teuthology.task.internal:Creating test directory... 2021-09-02T07:19:57.340 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-09-02T07:19:57.343 DEBUG:teuthology.orchestra.run.smithi197:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-09-02T07:19:57.357 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-09-02T07:19:57.388 INFO:teuthology.run_tasks:Running task internal.archive... 2021-09-02T07:19:57.421 INFO:teuthology.task.internal:Creating archive directory... 2021-09-02T07:19:57.422 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-09-02T07:19:57.425 DEBUG:teuthology.orchestra.run.smithi197:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-09-02T07:19:57.462 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-09-02T07:19:57.472 INFO:teuthology.task.internal:Enabling coredump saving... 2021-09-02T07:19:57.472 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-09-02T07:19:57.500 DEBUG:teuthology.orchestra.run.smithi197:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-09-02T07:19:57.532 INFO:teuthology.orchestra.run.smithi175.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-02T07:19:57.543 INFO:teuthology.orchestra.run.smithi175.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-02T07:19:57.544 INFO:teuthology.orchestra.run.smithi197.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-02T07:19:57.555 INFO:teuthology.orchestra.run.smithi197.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-02T07:19:57.556 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-09-02T07:19:57.567 INFO:teuthology.task.internal:Configuring sudo... 2021-09-02T07:19:57.568 DEBUG:teuthology.orchestra.run.smithi175:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-09-02T07:19:57.589 DEBUG:teuthology.orchestra.run.smithi197:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-09-02T07:19:57.615 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-09-02T07:19:57.625 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-09-02T07:19:57.626 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-09-02T07:19:57.655 DEBUG:teuthology.orchestra.run.smithi197:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-09-02T07:19:57.669 DEBUG:teuthology.orchestra.run.smithi175:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-02T07:19:57.744 DEBUG:teuthology.orchestra.run.smithi175:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-02T07:19:57.788 DEBUG:teuthology.orchestra.run.smithi175:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-02T07:19:57.863 DEBUG:teuthology.orchestra.run.smithi175:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-02T07:19:57.910 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:19:57.911 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-09-02T07:19:57.973 DEBUG:teuthology.orchestra.run.smithi197:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-02T07:19:58.011 DEBUG:teuthology.orchestra.run.smithi197:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-02T07:19:58.057 DEBUG:teuthology.orchestra.run.smithi197:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-02T07:19:58.127 DEBUG:teuthology.orchestra.run.smithi197:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-02T07:19:58.170 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:19:58.170 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-09-02T07:19:58.231 DEBUG:teuthology.orchestra.run.smithi175:> sudo service rsyslog restart 2021-09-02T07:19:58.233 DEBUG:teuthology.orchestra.run.smithi197:> sudo service rsyslog restart 2021-09-02T07:19:58.278 INFO:teuthology.orchestra.run.smithi175.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-09-02T07:19:58.337 INFO:teuthology.orchestra.run.smithi197.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-09-02T07:19:58.674 INFO:teuthology.run_tasks:Running task internal.timer... 2021-09-02T07:19:58.690 INFO:teuthology.task.internal:Starting timer... 2021-09-02T07:19:58.691 INFO:teuthology.run_tasks:Running task pcp... 2021-09-02T07:19:58.744 INFO:teuthology.run_tasks:Running task selinux... 2021-09-02T07:19:58.773 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-09-02T07:19:58.774 DEBUG:teuthology.orchestra.run.smithi175:> sudo service auditd rotate 2021-09-02T07:19:58.828 INFO:teuthology.orchestra.run.smithi175.stdout:Rotating logs: [ OK ] 2021-09-02T07:19:58.830 DEBUG:teuthology.orchestra.run.smithi197:> sudo service auditd rotate 2021-09-02T07:19:58.899 INFO:teuthology.orchestra.run.smithi197.stdout:Rotating logs: [ OK ] 2021-09-02T07:19:58.900 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-09-02T07:19:58.901 DEBUG:teuthology.orchestra.run.smithi175:> /usr/sbin/getenforce 2021-09-02T07:19:59.164 INFO:teuthology.orchestra.run.smithi175.stdout:Permissive 2021-09-02T07:19:59.165 DEBUG:teuthology.orchestra.run.smithi197:> /usr/sbin/getenforce 2021-09-02T07:19:59.188 INFO:teuthology.orchestra.run.smithi197.stdout:Permissive 2021-09-02T07:19:59.189 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi175.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi197.front.sepia.ceph.com': 'permissive'} 2021-09-02T07:19:59.189 DEBUG:teuthology.orchestra.run.smithi175:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-09-02T07:19:59.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-02T07:19:59.217 DEBUG:teuthology.orchestra.run.smithi197:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-09-02T07:19:59.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-02T07:19:59.274 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-09-02T07:19:59.274 DEBUG:teuthology.orchestra.run.smithi175:> sudo /usr/sbin/setenforce permissive 2021-09-02T07:19:59.312 DEBUG:teuthology.orchestra.run.smithi197:> sudo /usr/sbin/setenforce permissive 2021-09-02T07:19:59.369 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-09-02T07:19:59.393 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-09-02T07:19:59.401 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-09-02T07:19:59.424 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2021-09-02T07:19:59.425 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi175.front.sepia.ceph.com,smithi197.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-09-02T07:19:59.801 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-09-02T07:20:00.618 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-09-02T07:20:03.509 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-09-02T07:20:03.530 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:03 +0000 (0:00:00.041) 0:00:00.041 **** 2021-09-02T07:20:03.542 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:03 +0000 (0:00:00.011) 0:00:00.052 **** 2021-09-02T07:20:05.019 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi175.front.sepia.ceph.com] 2021-09-02T07:20:05.080 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:05 +0000 (0:00:01.538) 0:00:01.591 **** 2021-09-02T07:20:05.352 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-09-02T07:20:05.388 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:05 +0000 (0:00:00.308) 0:00:01.899 **** 2021-09-02T07:20:05.760 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-09-02T07:20:05.761 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-09-02T07:20:05.783 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:05 +0000 (0:00:00.394) 0:00:02.293 **** 2021-09-02T07:20:05.808 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi197.front.sepia.ceph.com] 2021-09-02T07:20:05.857 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:05 +0000 (0:00:00.074) 0:00:02.367 **** 2021-09-02T07:20:05.864 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-09-02T07:20:05.865 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:20:05.891 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:05 +0000 (0:00:00.034) 0:00:02.402 **** 2021-09-02T07:20:06.045 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-09-02T07:20:06.047 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-09-02T07:20:06.067 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:06 +0000 (0:00:00.176) 0:00:02.578 **** 2021-09-02T07:20:06.168 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-09-02T07:20:06.170 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-02T07:20:06.192 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:06 +0000 (0:00:00.124) 0:00:02.702 **** 2021-09-02T07:20:06.357 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-09-02T07:20:06.359 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-09-02T07:20:06.379 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:06 +0000 (0:00:00.187) 0:00:02.890 **** 2021-09-02T07:20:06.448 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-09-02T07:20:06.450 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:20:06.470 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:06 +0000 (0:00:00.090) 0:00:02.981 **** 2021-09-02T07:20:06.664 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi197.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-02T07:20:06.686 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:06 +0000 (0:00:00.215) 0:00:03.196 **** 2021-09-02T07:20:07.063 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1630567206.239458-20117-39504959485153/source", "state": "file", "uid": 0} 2021-09-02T07:20:07.086 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:07 +0000 (0:00:00.399) 0:00:03.596 **** 2021-09-02T07:20:07.299 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1630567206.7294445-20187-124888536342862/source", "state": "file", "uid": 0} 2021-09-02T07:20:07.326 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:07 +0000 (0:00:00.239) 0:00:03.836 **** 2021-09-02T07:20:07.527 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-09-02T07:20:07.529 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-09-02T07:20:07.530 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-09-02T07:20:07.657 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-09-02T07:20:07.659 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-09-02T07:20:07.720 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-09-02T07:20:07.721 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-09-02T07:20:07.735 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-09-02T07:20:07.736 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:07 +0000 (0:00:00.409) 0:00:04.245 **** 2021-09-02T07:20:08.244 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] 2021-09-02T07:20:08.263 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] 2021-09-02T07:20:08.280 INFO:teuthology.task.ansible.out: TASK [set_fact] **************************************************************** Thursday 02 September 2021 07:20:08 +0000 (0:00:00.545) 0:00:04.791 **** 2021-09-02T07:20:08.305 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-09-02T07:20:08.316 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-09-02T07:20:08.322 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-09-02T07:20:08.479 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:08 +0000 (0:00:00.198) 0:00:04.989 **** 2021-09-02T07:20:08.499 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:08 +0000 (0:00:00.020) 0:00:05.009 **** 2021-09-02T07:20:09.122 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi197.front.sepia.ceph.com] 2021-09-02T07:20:09.176 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-09-02T07:20:09.178 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:09 +0000 (0:00:00.676) 0:00:05.686 **** 2021-09-02T07:20:09.845 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-09-02T07:20:09.846 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.173495", "end": "2021-09-02 07:20:09.816979", "rc": 0, "start": "2021-09-02 07:20:09.643484", "stderr": "", "stderr_lines": [], "stdout": "540ef3733cffec190e79eb6099e95cef354b655b", "stdout_lines": ["540ef3733cffec190e79eb6099e95cef354b655b"]} 2021-09-02T07:20:09.867 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:09 +0000 (0:00:00.690) 0:00:06.377 **** 2021-09-02T07:20:10.072 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-09-02T07:20:10.073 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.008134", "end": "2021-09-02 07:20:10.047119", "failed_when_result": false, "rc": 0, "start": "2021-09-02 07:20:10.038985", "stderr": "", "stderr_lines": [], "stdout": "c29274177b79f72740fc714fa28b55400cd0737f", "stdout_lines": ["c29274177b79f72740fc714fa28b55400cd0737f"]} 2021-09-02T07:20:10.094 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:10 +0000 (0:00:00.226) 0:00:06.604 **** 2021-09-02T07:20:10.114 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi175.front.sepia.ceph.com] 2021-09-02T07:20:10.166 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:10 +0000 (0:00:00.072) 0:00:06.676 **** 2021-09-02T07:20:10.173 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-09-02T07:20:10.175 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:20:10.199 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:10 +0000 (0:00:00.032) 0:00:06.709 **** 2021-09-02T07:20:10.281 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:20:10.304 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:10 +0000 (0:00:00.104) 0:00:06.814 **** 2021-09-02T07:20:10.391 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-09-02T07:20:10.393 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:20:10.416 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:10 +0000 (0:00:00.112) 0:00:06.926 **** 2021-09-02T07:20:10.497 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:20:10.521 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:10 +0000 (0:00:00.104) 0:00:07.031 **** 2021-09-02T07:20:10.593 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:20:10.615 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:10 +0000 (0:00:00.094) 0:00:07.126 **** 2021-09-02T07:20:10.623 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.191025", "end": "2021-09-02 07:20:10.587642", "rc": 0, "start": "2021-09-02 07:20:10.396617", "stderr": "", "stderr_lines": [], "stdout": "540ef3733cffec190e79eb6099e95cef354b655b", "stdout_lines": ["540ef3733cffec190e79eb6099e95cef354b655b"]} 2021-09-02T07:20:10.647 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:10 +0000 (0:00:00.031) 0:00:07.157 **** 2021-09-02T07:20:10.840 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-09-02T07:20:10.842 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-02T07:20:10.872 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:10 +0000 (0:00:00.224) 0:00:07.382 **** 2021-09-02T07:20:10.883 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.009142", "end": "2021-09-02 07:20:10.825270", "failed_when_result": false, "rc": 0, "start": "2021-09-02 07:20:10.816128", "stderr": "", "stderr_lines": [], "stdout": "c29274177b79f72740fc714fa28b55400cd0737f", "stdout_lines": ["c29274177b79f72740fc714fa28b55400cd0737f"]} 2021-09-02T07:20:10.907 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:10 +0000 (0:00:00.035) 0:00:07.418 **** 2021-09-02T07:20:10.979 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-09-02T07:20:10.981 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:20:11.001 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:10 +0000 (0:00:00.094) 0:00:07.512 **** 2021-09-02T07:20:11.064 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-09-02T07:20:11.066 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:20:11.087 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:11 +0000 (0:00:00.084) 0:00:07.596 **** 2021-09-02T07:20:11.154 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-09-02T07:20:11.157 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-09-02T07:20:11.166 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:20:11.189 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:11 +0000 (0:00:00.102) 0:00:07.699 **** 2021-09-02T07:20:11.250 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:20:11.272 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:11 +0000 (0:00:00.083) 0:00:07.783 **** 2021-09-02T07:20:11.343 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:20:11.344 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-09-02T07:20:11.366 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:11 +0000 (0:00:00.093) 0:00:07.876 **** 2021-09-02T07:20:11.542 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-09-02T07:20:11.606 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-09-02T07:20:11.608 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-02T07:20:11.629 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:11 +0000 (0:00:00.262) 0:00:08.139 **** 2021-09-02T07:20:11.735 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-09-02T07:20:11.737 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-09-02T07:20:11.916 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-09-02T07:20:11.927 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-09-02T07:20:12.107 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-09-02T07:20:12.129 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-09-02T07:20:12.301 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-09-02T07:20:12.313 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-09-02T07:20:12.503 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-09-02T07:20:12.516 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-09-02T07:20:12.695 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-09-02T07:20:12.706 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-09-02T07:20:12.886 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-09-02T07:20:12.898 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-09-02T07:20:13.088 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-09-02T07:20:13.100 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-09-02T07:20:13.271 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-09-02T07:20:13.303 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-09-02T07:20:13.471 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-09-02T07:20:13.494 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-09-02T07:20:13.662 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-09-02T07:20:13.694 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-09-02T07:20:13.860 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-09-02T07:20:13.893 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-09-02T07:20:14.049 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-09-02T07:20:14.081 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-09-02T07:20:14.249 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-09-02T07:20:14.281 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-09-02T07:20:14.448 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-09-02T07:20:14.471 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-09-02T07:20:14.640 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-09-02T07:20:14.672 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-09-02T07:20:14.832 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-09-02T07:20:14.875 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-09-02T07:20:15.036 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-09-02T07:20:15.068 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-09-02T07:20:15.224 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-09-02T07:20:15.266 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-09-02T07:20:15.419 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-09-02T07:20:15.463 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-09-02T07:20:15.622 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-09-02T07:20:15.654 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-09-02T07:20:15.816 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-09-02T07:20:15.850 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-09-02T07:20:16.019 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-09-02T07:20:16.052 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-09-02T07:20:16.210 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-09-02T07:20:16.254 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-09-02T07:20:16.412 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-09-02T07:20:16.456 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-09-02T07:20:16.606 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-09-02T07:20:16.650 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-09-02T07:20:16.818 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-09-02T07:20:16.840 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-09-02T07:20:17.010 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-09-02T07:20:17.035 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-09-02T07:20:17.203 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-09-02T07:20:17.235 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-09-02T07:20:17.403 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-09-02T07:20:17.435 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-09-02T07:20:17.602 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-09-02T07:20:17.631 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-09-02T07:20:17.795 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-09-02T07:20:17.827 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-09-02T07:20:17.994 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-09-02T07:20:18.017 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-09-02T07:20:18.184 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-09-02T07:20:18.219 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-09-02T07:20:18.387 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-09-02T07:20:18.419 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-02T07:20:18.577 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-09-02T07:20:18.621 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-09-02T07:20:18.777 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-09-02T07:20:18.811 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-09-02T07:20:18.970 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-09-02T07:20:19.003 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-09-02T07:20:19.171 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-02T07:20:19.205 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-09-02T07:20:19.364 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-09-02T07:20:19.396 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-09-02T07:20:19.564 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-09-02T07:20:19.596 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-09-02T07:20:19.753 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-09-02T07:20:19.798 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-09-02T07:20:19.956 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-09-02T07:20:19.988 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-02T07:20:20.147 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-09-02T07:20:20.191 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-09-02T07:20:20.339 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-09-02T07:20:20.382 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-09-02T07:20:20.540 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-09-02T07:20:20.594 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-09-02T07:20:20.734 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-02T07:20:20.787 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-09-02T07:20:20.935 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-09-02T07:20:20.991 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-09-02T07:20:21.126 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-09-02T07:20:21.204 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-09-02T07:20:21.334 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-09-02T07:20:21.394 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-09-02T07:20:21.528 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-09-02T07:20:21.581 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-09-02T07:20:21.731 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-09-02T07:20:21.781 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-09-02T07:20:21.929 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-09-02T07:20:21.971 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-09-02T07:20:22.119 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-09-02T07:20:22.165 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-09-02T07:20:22.312 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-09-02T07:20:22.365 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-09-02T07:20:22.512 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-09-02T07:20:22.555 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-09-02T07:20:22.703 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-09-02T07:20:22.756 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-09-02T07:20:22.902 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-09-02T07:20:22.945 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-09-02T07:20:23.092 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-09-02T07:20:23.145 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-09-02T07:20:23.284 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-09-02T07:20:23.337 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-02T07:20:23.485 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-09-02T07:20:23.539 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-09-02T07:20:23.675 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-09-02T07:20:23.738 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-09-02T07:20:23.875 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-09-02T07:20:23.918 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-09-02T07:20:24.067 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-02T07:20:24.122 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-09-02T07:20:24.258 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-09-02T07:20:24.322 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-09-02T07:20:24.449 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-09-02T07:20:24.514 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-09-02T07:20:24.650 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-09-02T07:20:24.705 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-09-02T07:20:24.853 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-09-02T07:20:24.906 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-09-02T07:20:25.043 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-09-02T07:20:25.098 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-09-02T07:20:25.236 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-09-02T07:20:25.290 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-09-02T07:20:25.441 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-09-02T07:20:25.485 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-09-02T07:20:25.632 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-09-02T07:20:25.687 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-09-02T07:20:25.823 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-09-02T07:20:25.876 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-09-02T07:20:26.024 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-09-02T07:20:26.066 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-09-02T07:20:26.213 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-09-02T07:20:26.266 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-09-02T07:20:26.416 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-09-02T07:20:26.461 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-09-02T07:20:26.614 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-09-02T07:20:26.667 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-09-02T07:20:26.814 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-09-02T07:20:26.857 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-09-02T07:20:27.008 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-09-02T07:20:27.050 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-09-02T07:20:27.198 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-09-02T07:20:27.248 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-09-02T07:20:27.390 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-09-02T07:20:27.443 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-09-02T07:20:27.591 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-09-02T07:20:27.644 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-09-02T07:20:27.793 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-09-02T07:20:27.836 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-09-02T07:20:27.983 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-09-02T07:20:28.038 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-09-02T07:20:28.185 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-09-02T07:20:28.232 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-09-02T07:20:28.380 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-09-02T07:20:28.434 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-09-02T07:20:28.572 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-09-02T07:20:28.636 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-09-02T07:20:28.764 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-09-02T07:20:28.819 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-09-02T07:20:28.957 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-09-02T07:20:29.020 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-09-02T07:20:29.157 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-09-02T07:20:29.210 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-09-02T07:20:29.353 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-09-02T07:20:29.406 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-09-02T07:20:29.545 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-09-02T07:20:29.608 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-09-02T07:20:29.745 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-09-02T07:20:29.798 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-09-02T07:20:29.947 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-09-02T07:20:29.992 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-09-02T07:20:30.130 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-09-02T07:20:30.196 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-09-02T07:20:30.329 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-09-02T07:20:30.408 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-09-02T07:20:30.558 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-09-02T07:20:30.688 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-09-02T07:20:30.762 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-09-02T07:20:30.945 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-02T07:20:30.979 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-09-02T07:20:31.167 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-09-02T07:20:31.201 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-09-02T07:20:31.384 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-09-02T07:20:31.409 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-09-02T07:20:31.589 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-09-02T07:20:31.601 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-02T07:20:31.778 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-09-02T07:20:31.800 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-09-02T07:20:31.978 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-09-02T07:20:31.989 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-09-02T07:20:32.173 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-09-02T07:20:32.195 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-09-02T07:20:32.365 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-09-02T07:20:32.388 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-09-02T07:20:32.566 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-09-02T07:20:32.600 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-09-02T07:20:32.770 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-09-02T07:20:32.792 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-09-02T07:20:32.963 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-09-02T07:20:32.996 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-09-02T07:20:33.154 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-09-02T07:20:33.186 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-09-02T07:20:33.355 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-09-02T07:20:33.387 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-09-02T07:20:33.546 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-09-02T07:20:33.589 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-09-02T07:20:33.747 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-09-02T07:20:33.780 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-09-02T07:20:33.938 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-09-02T07:20:33.973 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-09-02T07:20:34.142 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-09-02T07:20:34.175 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-09-02T07:20:34.333 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-09-02T07:20:34.367 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-09-02T07:20:34.534 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-09-02T07:20:34.569 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-09-02T07:20:34.726 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-09-02T07:20:34.768 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-09-02T07:20:34.915 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-09-02T07:20:34.969 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-09-02T07:20:35.116 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-09-02T07:20:35.160 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-09-02T07:20:35.307 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-09-02T07:20:35.362 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-09-02T07:20:35.503 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-09-02T07:20:35.556 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-09-02T07:20:35.711 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-09-02T07:20:35.750 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-09-02T07:20:35.898 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-09-02T07:20:35.951 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-09-02T07:20:36.088 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-09-02T07:20:36.142 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-09-02T07:20:36.289 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-09-02T07:20:36.331 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-09-02T07:20:36.480 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-09-02T07:20:36.534 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-09-02T07:20:36.681 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-09-02T07:20:36.724 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-09-02T07:20:36.902 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-09-02T07:20:36.913 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-09-02T07:20:37.093 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-09-02T07:20:37.115 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-09-02T07:20:37.284 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-09-02T07:20:37.306 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-09-02T07:20:37.488 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-09-02T07:20:37.521 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-09-02T07:20:37.679 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-09-02T07:20:37.722 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-09-02T07:20:37.879 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-09-02T07:20:37.924 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-09-02T07:20:38.071 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-09-02T07:20:38.115 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-09-02T07:20:38.273 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-09-02T07:20:38.316 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-09-02T07:20:38.464 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-09-02T07:20:38.517 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-09-02T07:20:38.663 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1238, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "move_home": false, "name": "amctagga", "shell": "/bin/bash", "state": "present", "uid": 1238} 2021-09-02T07:20:38.706 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-09-02T07:20:38.853 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1239, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "move_home": false, "name": "kevinz", "shell": "/bin/bash", "state": "present", "uid": 1239} 2021-09-02T07:20:38.906 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-09-02T07:20:39.054 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1240, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "move_home": false, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "uid": 1240} 2021-09-02T07:20:39.107 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-09-02T07:20:39.248 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1241, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "move_home": false, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "uid": 1241} 2021-09-02T07:20:39.303 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1238, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "move_home": false, "name": "amctagga", "shell": "/bin/bash", "state": "present", "uid": 1238} 2021-09-02T07:20:39.442 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1242, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "move_home": false, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "uid": 1242} 2021-09-02T07:20:39.506 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1239, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "move_home": false, "name": "kevinz", "shell": "/bin/bash", "state": "present", "uid": 1239} 2021-09-02T07:20:39.634 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1243, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "move_home": false, "name": "lflores", "shell": "/bin/bash", "state": "present", "uid": 1243} 2021-09-02T07:20:39.697 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1240, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "move_home": false, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "uid": 1240} 2021-09-02T07:20:39.833 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1244, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "move_home": false, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "uid": 1244} 2021-09-02T07:20:39.896 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1241, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "move_home": false, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "uid": 1241} 2021-09-02T07:20:40.044 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1245, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "move_home": false, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "uid": 1245} 2021-09-02T07:20:40.098 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1242, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "move_home": false, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "uid": 1242} 2021-09-02T07:20:40.234 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1246, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "move_home": false, "name": "luisrico", "shell": "/bin/bash", "state": "present", "uid": 1246} 2021-09-02T07:20:40.298 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1243, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "move_home": false, "name": "lflores", "shell": "/bin/bash", "state": "present", "uid": 1243} 2021-09-02T07:20:40.432 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1247, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "move_home": false, "name": "matan", "shell": "/bin/bash", "state": "present", "uid": 1247} 2021-09-02T07:20:40.495 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1244, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "move_home": false, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "uid": 1244} 2021-09-02T07:20:40.632 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1248, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "move_home": false, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "uid": 1248} 2021-09-02T07:20:40.698 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1245, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "move_home": false, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "uid": 1245} 2021-09-02T07:20:40.823 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1249, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "move_home": false, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "uid": 1249} 2021-09-02T07:20:40.888 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1246, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "move_home": false, "name": "luisrico", "shell": "/bin/bash", "state": "present", "uid": 1246} 2021-09-02T07:20:41.014 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1250, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "move_home": false, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "uid": 1250} 2021-09-02T07:20:41.087 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1247, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "move_home": false, "name": "matan", "shell": "/bin/bash", "state": "present", "uid": 1247} 2021-09-02T07:20:41.212 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1251, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "move_home": false, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "uid": 1251} 2021-09-02T07:20:41.297 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1248, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "move_home": false, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "uid": 1248} 2021-09-02T07:20:41.408 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1252, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "move_home": false, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "uid": 1252} 2021-09-02T07:20:41.493 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1249, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "move_home": false, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "uid": 1249} 2021-09-02T07:20:41.609 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1253, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "move_home": false, "name": "melissali", "shell": "/bin/bash", "state": "present", "uid": 1253} 2021-09-02T07:20:41.684 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1250, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "move_home": false, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "uid": 1250} 2021-09-02T07:20:41.884 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1251, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "move_home": false, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "uid": 1251} 2021-09-02T07:20:42.074 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1252, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "move_home": false, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "uid": 1252} 2021-09-02T07:20:42.273 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1253, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "move_home": false, "name": "melissali", "shell": "/bin/bash", "state": "present", "uid": 1253} 2021-09-02T07:20:43.330 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-09-02T07:20:43.927 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-09-02T07:20:45.077 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-09-02T07:20:45.675 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-09-02T07:20:46.659 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-09-02T07:20:46.809 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:46 +0000 (0:00:35.179) 0:00:43.319 **** 2021-09-02T07:20:46.881 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-09-02T07:20:46.908 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:46 +0000 (0:00:00.094) 0:00:43.414 **** 2021-09-02T07:20:46.991 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** ok: [smithi197.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c 2021-09-02T07:20:46.993 INFO:teuthology.task.ansible.out:68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "a 2021-09-02T07:20:46.994 INFO:teuthology.task.ansible.out:li@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb27 2021-09-02T07:20:46.996 INFO:teuthology.task.ansible.out:3afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767 2021-09-02T07:20:46.998 INFO:teuthology.task.ansible.out:dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99 2021-09-02T07:20:46.998 INFO:teuthology.task.ansible.out:bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name" 2021-09-02T07:20:47.000 INFO:teuthology.task.ansible.out:: "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42 2021-09-02T07:20:47.002 INFO:teuthology.task.ansible.out:c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3 2021-09-02T07:20:47.004 INFO:teuthology.task.ansible.out:ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf 2021-09-02T07:20:47.006 INFO:teuthology.task.ansible.out:2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3 2021-09-02T07:20:47.007 INFO:teuthology.task.ansible.out:f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQ 2021-09-02T07:20:47.009 INFO:teuthology.task.ansible.out:PrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "pr 2021-09-02T07:20:47.011 INFO:teuthology.task.ansible.out:ateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf 2021-09-02T07:20:47.012 INFO:teuthology.task.ansible.out:8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": " 2021-09-02T07:20:47.014 INFO:teuthology.task.ansible.out:cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d 2021-09-02T07:20:47.016 INFO:teuthology.task.ansible.out:56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}]}, "changed": false} 2021-09-02T07:20:47.036 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:47 +0000 (0:00:00.131) 0:00:43.546 **** 2021-09-02T07:20:47.267 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-09-02T07:20:47.269 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-09-02T07:20:47.449 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:47 +0000 (0:00:00.413) 0:00:43.959 **** 2021-09-02T07:20:47.517 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-09-02T07:20:47.539 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:47 +0000 (0:00:00.090) 0:00:44.049 **** 2021-09-02T07:20:47.633 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-09-02T07:20:47.635 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-09-02T07:20:47.637 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-09-02T07:20:47.638 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-09-02T07:20:47.640 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-09-02T07:20:47.642 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9 2021-09-02T07:20:47.644 INFO:teuthology.task.ansible.out:b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", " 2021-09-02T07:20:47.646 INFO:teuthology.task.ansible.out:ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn" 2021-09-02T07:20:47.647 INFO:teuthology.task.ansible.out:: "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw 2021-09-02T07:20:47.649 INFO:teuthology.task.ansible.out: b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {" 2021-09-02T07:20:47.650 INFO:teuthology.task.ansible.out:name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr 2021-09-02T07:20:47.655 INFO:teuthology.task.ansible.out:@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e104 2021-09-02T07:20:47.656 INFO:teuthology.task.ansible.out:6655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": 2021-09-02T07:20:47.658 INFO:teuthology.task.ansible.out: "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809 2021-09-02T07:20:47.660 INFO:teuthology.task.ansible.out:f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmu 2021-09-02T07:20:47.662 INFO:teuthology.task.ansible.out:njulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b 2021-09-02T07:20:47.664 INFO:teuthology.task.ansible.out:8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}]}, "changed": false} 2021-09-02T07:20:47.682 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:47 +0000 (0:00:00.143) 0:00:44.192 **** 2021-09-02T07:20:48.010 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-09-02T07:20:48.012 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"after": "540ef3733cffec190e79eb6099e95cef354b655b", "attempts": 1, "before": "540ef3733cffec190e79eb6099e95cef354b655b", "changed": false, "remote_url_changed": false} 2021-09-02T07:20:48.035 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:48 +0000 (0:00:00.353) 0:00:44.546 **** 2021-09-02T07:20:48.359 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"after": "540ef3733cffec190e79eb6099e95cef354b655b", "attempts": 1, "before": "540ef3733cffec190e79eb6099e95cef354b655b", "changed": false, "remote_url_changed": false} 2021-09-02T07:20:48.378 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:20:48 +0000 (0:00:00.342) 0:00:44.888 **** 2021-09-02T07:20:48.388 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-09-02T07:20:48.390 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-09-02T07:20:48.581 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-09-02T07:20:48.712 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-09-02T07:20:48.805 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-09-02T07:20:48.933 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-09-02T07:20:49.023 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-09-02T07:20:49.024 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-09-02T07:20:49.139 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-09-02T07:20:49.237 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-09-02T07:20:49.343 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-09-02T07:20:49.345 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-09-02T07:20:49.448 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-09-02T07:20:49.450 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-09-02T07:20:49.564 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-09-02T07:20:49.677 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-09-02T07:20:49.763 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-09-02T07:20:49.765 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-09-02T07:20:49.878 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-09-02T07:20:49.963 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-09-02T07:20:50.081 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-09-02T07:20:50.093 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:20:50.167 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-09-02T07:20:50.304 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-09-02T07:20:50.306 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-09-02T07:20:50.369 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-09-02T07:20:50.380 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:20:50.516 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-09-02T07:20:50.518 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-09-02T07:20:50.527 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:20:50.538 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:20:50.591 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-09-02T07:20:50.593 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-09-02T07:20:50.738 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-09-02T07:20:50.792 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-09-02T07:20:50.794 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-09-02T07:20:50.803 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:20:50.815 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:20:50.943 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-09-02T07:20:50.945 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-09-02T07:20:51.017 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-09-02T07:20:51.153 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-09-02T07:20:51.219 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-09-02T07:20:51.220 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-09-02T07:20:51.362 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-09-02T07:20:51.364 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-09-02T07:20:51.436 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-09-02T07:20:51.565 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-09-02T07:20:51.639 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-09-02T07:20:51.641 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-09-02T07:20:51.778 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-09-02T07:20:51.851 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-09-02T07:20:52.001 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-09-02T07:20:52.064 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-09-02T07:20:52.204 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-09-02T07:20:52.206 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-09-02T07:20:52.270 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-09-02T07:20:52.407 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-09-02T07:20:52.409 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-09-02T07:20:52.473 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-09-02T07:20:52.474 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-09-02T07:20:52.621 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-09-02T07:20:52.690 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-09-02T07:20:52.701 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-09-02T07:20:52.835 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-09-02T07:20:52.888 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-09-02T07:20:53.040 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlI 2021-09-02T07:20:53.042 INFO:teuthology.task.ansible.out:Qv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-09-02T07:20:53.104 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-09-02T07:20:53.250 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-09-02T07:20:53.303 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlI 2021-09-02T07:20:53.305 INFO:teuthology.task.ansible.out:Qv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-09-02T07:20:53.450 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-09-02T07:20:53.514 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-09-02T07:20:53.653 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-09-02T07:20:53.717 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-09-02T07:20:53.865 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-09-02T07:20:53.931 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-09-02T07:20:54.070 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-09-02T07:20:54.134 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-09-02T07:20:54.281 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-09-02T07:20:54.283 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-09-02T07:20:54.345 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-09-02T07:20:54.483 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-09-02T07:20:54.556 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-09-02T07:20:54.558 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-09-02T07:20:54.700 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-09-02T07:20:54.763 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-09-02T07:20:54.913 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-09-02T07:20:54.977 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-09-02T07:20:55.114 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-09-02T07:20:55.190 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-09-02T07:20:55.328 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-09-02T07:20:55.391 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-09-02T07:20:55.529 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-02T07:20:55.597 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-09-02T07:20:55.739 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-09-02T07:20:55.804 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-02T07:20:55.942 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-09-02T07:20:56.018 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-09-02T07:20:56.154 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-09-02T07:20:56.156 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-09-02T07:20:56.219 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-09-02T07:20:56.372 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-09-02T07:20:56.374 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-09-02T07:20:56.451 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-09-02T07:20:56.453 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-09-02T07:20:56.578 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-09-02T07:20:56.662 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-09-02T07:20:56.664 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-09-02T07:20:56.790 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-09-02T07:20:56.862 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-09-02T07:20:56.988 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-09-02T07:20:57.075 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-09-02T07:20:57.203 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-09-02T07:20:57.205 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-02T07:20:57.278 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-09-02T07:20:57.418 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-09-02T07:20:57.420 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-09-02T07:20:57.422 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-09-02T07:20:57.495 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-09-02T07:20:57.497 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-02T07:20:57.628 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-09-02T07:20:57.703 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-09-02T07:20:57.705 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-09-02T07:20:57.707 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-09-02T07:20:57.841 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-09-02T07:20:57.920 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-09-02T07:20:58.059 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-09-02T07:20:58.135 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-09-02T07:20:58.260 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-09-02T07:20:58.355 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-09-02T07:20:58.472 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-09-02T07:20:58.556 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-09-02T07:20:58.673 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-09-02T07:20:58.674 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-09-02T07:20:58.757 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-09-02T07:20:58.875 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-09-02T07:20:58.960 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-09-02T07:20:58.962 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-09-02T07:20:59.092 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-09-02T07:20:59.166 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-09-02T07:20:59.304 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-09-02T07:20:59.306 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-09-02T07:20:59.378 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-09-02T07:20:59.511 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-09-02T07:20:59.578 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-09-02T07:20:59.579 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-09-02T07:20:59.728 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-09-02T07:20:59.795 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-09-02T07:20:59.931 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-09-02T07:20:59.938 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-09-02T07:21:00.009 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-09-02T07:21:00.156 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-09-02T07:21:00.241 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-09-02T07:21:00.243 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-09-02T07:21:00.379 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-09-02T07:21:00.445 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-09-02T07:21:00.592 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-09-02T07:21:00.658 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-09-02T07:21:00.797 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-02T07:21:00.860 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-09-02T07:21:00.997 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-09-02T07:21:01.061 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-02T07:21:01.213 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-09-02T07:21:01.272 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-09-02T07:21:01.450 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-09-02T07:21:01.537 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-09-02T07:21:01.691 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-09-02T07:21:01.755 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-09-02T07:21:01.936 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-09-02T07:21:02.021 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-09-02T07:21:02.152 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-09-02T07:21:02.259 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-09-02T07:21:02.373 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-09-02T07:21:02.469 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-09-02T07:21:02.585 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-09-02T07:21:02.680 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-09-02T07:21:02.785 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-09-02T07:21:02.882 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-09-02T07:21:02.999 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-09-02T07:21:03.223 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-09-02T07:21:03.234 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-09-02T07:21:03.418 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-09-02T07:21:03.460 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-09-02T07:21:03.645 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-09-02T07:21:03.674 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-09-02T07:21:03.848 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-09-02T07:21:03.889 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-09-02T07:21:04.057 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-09-02T07:21:04.102 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-09-02T07:21:04.332 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-09-02T07:21:04.419 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-09-02T07:21:04.544 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-09-02T07:21:04.619 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-09-02T07:21:04.621 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-09-02T07:21:04.753 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-09-02T07:21:04.858 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-09-02T07:21:04.966 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-09-02T07:21:04.969 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-09-02T07:21:05.081 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-09-02T07:21:05.188 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-09-02T07:21:05.292 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-09-02T07:21:05.387 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-09-02T07:21:05.494 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-09-02T07:21:05.588 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-09-02T07:21:05.704 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-09-02T07:21:05.706 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-09-02T07:21:05.800 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-09-02T07:21:05.915 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-09-02T07:21:05.917 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-09-02T07:21:06.011 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-09-02T07:21:06.013 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-09-02T07:21:06.129 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-09-02T07:21:06.215 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-09-02T07:21:06.217 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-09-02T07:21:06.331 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-09-02T07:21:06.416 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-09-02T07:21:06.544 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-09-02T07:21:06.628 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-09-02T07:21:06.745 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-09-02T07:21:06.830 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-09-02T07:21:06.948 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-09-02T07:21:07.041 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-09-02T07:21:07.168 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-09-02T07:21:07.241 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-09-02T07:21:07.378 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-09-02T07:21:07.452 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-09-02T07:21:07.580 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-09-02T07:21:07.582 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-09-02T07:21:07.654 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-09-02T07:21:07.792 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-09-02T07:21:07.867 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-09-02T07:21:07.868 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-09-02T07:21:08.003 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-09-02T07:21:08.077 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-09-02T07:21:08.204 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-09-02T07:21:08.279 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-09-02T07:21:08.422 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-09-02T07:21:08.496 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-09-02T07:21:08.623 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-09-02T07:21:08.714 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-09-02T07:21:08.841 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-09-02T07:21:08.927 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-09-02T07:21:09.053 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-09-02T07:21:09.054 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-02T07:21:09.140 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-09-02T07:21:09.247 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-09-02T07:21:09.342 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-09-02T07:21:09.344 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-02T07:21:09.460 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-09-02T07:21:09.544 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-09-02T07:21:09.674 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-09-02T07:21:09.770 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-09-02T07:21:09.886 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-09-02T07:21:09.972 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-09-02T07:21:10.089 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-09-02T07:21:10.175 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-09-02T07:21:10.292 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-09-02T07:21:10.387 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-09-02T07:21:10.499 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-09-02T07:21:10.583 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-09-02T07:21:10.710 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-09-02T07:21:10.795 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-09-02T07:21:10.912 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-09-02T07:21:10.996 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-09-02T07:21:11.113 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-09-02T07:21:11.200 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-09-02T07:21:11.334 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-09-02T07:21:11.431 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-09-02T07:21:11.548 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-09-02T07:21:11.646 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-09-02T07:21:11.772 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-09-02T07:21:11.846 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-09-02T07:21:11.974 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-09-02T07:21:12.048 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-09-02T07:21:12.185 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-09-02T07:21:12.259 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-09-02T07:21:12.386 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-09-02T07:21:12.471 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-09-02T07:21:12.598 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-09-02T07:21:12.684 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-09-02T07:21:12.814 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-09-02T07:21:12.889 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-09-02T07:21:13.029 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-09-02T07:21:13.031 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-09-02T07:21:13.104 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-09-02T07:21:13.242 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-09-02T07:21:13.319 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-09-02T07:21:13.320 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-09-02T07:21:13.461 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-09-02T07:21:13.528 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-09-02T07:21:13.664 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-09-02T07:21:13.765 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-09-02T07:21:13.879 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-09-02T07:21:13.982 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-09-02T07:21:14.089 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-09-02T07:21:14.211 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-09-02T07:21:14.311 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-09-02T07:21:14.407 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-09-02T07:21:14.535 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-09-02T07:21:14.623 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-09-02T07:21:14.750 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-09-02T07:21:14.826 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-09-02T07:21:14.956 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-09-02T07:21:15.032 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-09-02T07:21:15.171 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-09-02T07:21:15.247 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-09-02T07:21:15.373 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-09-02T07:21:15.462 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-09-02T07:21:15.579 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-09-02T07:21:15.664 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-09-02T07:21:15.791 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-09-02T07:21:15.877 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-09-02T07:21:15.994 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-09-02T07:21:16.090 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-09-02T07:21:16.196 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-09-02T07:21:16.294 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-09-02T07:21:16.420 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-09-02T07:21:16.518 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-09-02T07:21:16.636 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-09-02T07:21:16.638 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-09-02T07:21:16.747 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-09-02T07:21:16.860 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-09-02T07:21:16.948 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-09-02T07:21:16.949 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-09-02T07:21:17.067 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-09-02T07:21:17.169 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-09-02T07:21:17.274 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-09-02T07:21:17.369 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-09-02T07:21:17.482 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-09-02T07:21:17.584 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-09-02T07:21:17.701 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-09-02T07:21:17.788 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-09-02T07:21:17.908 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-09-02T07:21:17.992 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-09-02T07:21:18.110 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-09-02T07:21:18.205 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-09-02T07:21:18.320 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-09-02T07:21:18.405 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-09-02T07:21:18.522 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-09-02T07:21:18.616 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-09-02T07:21:18.732 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-09-02T07:21:18.817 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-09-02T07:21:18.935 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-09-02T07:21:19.029 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-09-02T07:21:19.136 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-09-02T07:21:19.230 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-09-02T07:21:19.346 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-09-02T07:21:19.441 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-09-02T07:21:19.547 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-09-02T07:21:19.654 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-09-02T07:21:19.761 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-09-02T07:21:19.846 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-09-02T07:21:19.962 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-09-02T07:21:20.058 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-09-02T07:21:20.163 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-09-02T07:21:20.258 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-09-02T07:21:20.375 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-09-02T07:21:20.472 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-09-02T07:21:20.587 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-09-02T07:21:20.693 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-09-02T07:21:20.800 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-09-02T07:21:20.907 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-09-02T07:21:21.002 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-09-02T07:21:21.239 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-09-02T07:21:21.260 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:21 +0000 (0:00:32.881) 0:01:17.769 **** 2021-09-02T07:21:21.333 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-09-02T07:21:21.335 INFO:teuthology.task.ansible.out: 2021-09-02T07:21:21.413 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-09-02T07:21:21.414 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ 2021-09-02T07:21:21.416 INFO:teuthology.task.ansible.out:ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcN 2021-09-02T07:21:21.420 INFO:teuthology.task.ansible.out:E/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:21.438 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:21 +0000 (0:00:00.178) 0:01:17.948 **** 2021-09-02T07:21:21.532 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:21.543 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:21.554 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:21.556 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:21.566 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:21.577 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:21.578 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:21.588 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:21.600 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:21.611 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-09-02T07:21:21.622 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:21.633 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:21.802 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-09-02T07:21:21.813 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:21.825 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:21.836 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-09-02T07:21:22.025 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-09-02T07:21:22.027 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-09-02T07:21:22.037 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.048 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.059 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.071 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.073 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.084 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.086 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.095 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.097 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.107 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.118 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.129 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.132 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.141 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.152 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.152 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.163 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.165 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.174 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.185 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.188 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.196 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.198 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.208 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.219 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.220 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.230 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.231 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-09-02T07:21:22.232 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.242 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.246 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarn 2021-09-02T07:21:22.247 INFO:teuthology.task.ansible.out:um@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.255 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.256 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.267 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.269 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.278 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.280 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.290 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.292 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.302 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.304 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.306 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.306 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "o 2021-09-02T07:21:22.308 INFO:teuthology.task.ansible.out:vpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.318 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.319 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.329 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.331 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.341 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.343 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.354 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.356 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.367 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.368 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c 2021-09-02T07:21:22.370 INFO:teuthology.task.ansible.out:94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.380 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.381 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.392 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.394 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.404 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.406 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.418 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.420 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.430 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.432 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.443 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.445 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.455 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.466 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.479 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.481 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.490 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.491 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.492 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.503 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.506 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.507 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansibl 2021-09-02T07:21:22.509 INFO:teuthology.task.ansible.out:e_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.519 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.521 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.530 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.532 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.532 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.544 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.555 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.557 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.570 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.572 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.572 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.584 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.586 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.587 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.598 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.600 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.601 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.602 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.615 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.617 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.627 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.629 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.639 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.641 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.643 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.653 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.655 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.665 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.667 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.667 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.678 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.679 INFO:teuthology.task.ansible.out: 2021-09-02T07:21:22.681 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.692 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.695 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.703 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.705 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.717 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.719 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.740 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.743 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.753 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.754 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.755 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.755 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.756 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.756 INFO:teuthology.task.ansible.out:sk 2021-09-02T07:21:22.758 INFO:teuthology.task.ansible.out:ipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.771 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.774 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.785 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.788 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.797 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.799 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.810 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.812 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.823 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.824 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.834 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.836 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.846 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.848 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.859 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.861 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.871 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.873 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.873 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.876 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.885 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.889 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.899 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.901 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.923 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": 2021-09-02T07:21:22.926 INFO:teuthology.task.ansible.out: "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.935 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.937 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.938 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.939 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af 2021-09-02T07:21:22.941 INFO:teuthology.task.ansible.out:76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.950 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.962 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.964 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.964 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.975 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.977 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.987 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:22.989 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.000 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.002 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.023 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.024 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.025 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.025 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.053 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:23 +0000 (0:00:01.614) 0:01:19.563 **** 2021-09-02T07:21:23.060 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.062 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.074 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.075 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.076 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.087 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.089 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.098 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.109 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.111 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.121 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-09-02T07:21:23.124 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.144 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:23 +0000 (0:00:00.091) 0:01:19.654 **** 2021-09-02T07:21:23.151 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.152 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.154 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.166 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.177 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.179 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.188 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.189 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.200 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.221 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.223 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.233 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.244 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.246 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.256 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.257 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.308 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:23 +0000 (0:00:00.164) 0:01:19.819 **** 2021-09-02T07:21:23.378 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-09-02T07:21:23.380 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:23.401 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:23 +0000 (0:00:00.092) 0:01:19.911 **** 2021-09-02T07:21:23.419 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-09-02T07:21:23.421 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-09-02T07:21:23.611 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-09-02T07:21:23.676 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-09-02T07:21:23.804 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-09-02T07:21:23.867 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-09-02T07:21:23.994 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-09-02T07:21:24.068 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-09-02T07:21:24.189 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-09-02T07:21:24.264 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-09-02T07:21:24.390 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-09-02T07:21:24.454 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-09-02T07:21:24.570 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-09-02T07:21:24.645 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-09-02T07:21:24.765 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-09-02T07:21:24.839 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-09-02T07:21:24.955 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-09-02T07:21:25.030 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-09-02T07:21:25.155 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-09-02T07:21:25.220 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-09-02T07:21:25.347 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-09-02T07:21:25.413 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-09-02T07:21:25.532 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-09-02T07:21:25.617 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-09-02T07:21:25.734 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-09-02T07:21:25.799 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-09-02T07:21:25.925 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-09-02T07:21:25.998 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-09-02T07:21:26.113 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-09-02T07:21:26.187 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-09-02T07:21:26.302 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-09-02T07:21:26.377 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-09-02T07:21:26.493 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-09-02T07:21:26.567 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-09-02T07:21:26.693 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-09-02T07:21:26.756 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-09-02T07:21:26.884 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-09-02T07:21:26.958 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-09-02T07:21:27.064 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-09-02T07:21:27.150 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-09-02T07:21:27.267 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-09-02T07:21:27.340 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-09-02T07:21:27.456 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-09-02T07:21:27.540 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-09-02T07:21:27.644 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-09-02T07:21:27.722 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-09-02T07:21:27.837 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-09-02T07:21:27.918 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-09-02T07:21:28.032 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-09-02T07:21:28.117 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-09-02T07:21:28.223 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-09-02T07:21:28.306 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-09-02T07:21:28.422 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-09-02T07:21:28.495 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-09-02T07:21:28.611 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-09-02T07:21:28.685 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-09-02T07:21:28.800 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-09-02T07:21:28.874 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-09-02T07:21:28.995 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-09-02T07:21:29.070 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-09-02T07:21:29.186 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-09-02T07:21:29.260 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-09-02T07:21:29.379 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-09-02T07:21:29.453 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-09-02T07:21:29.580 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-09-02T07:21:29.644 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-09-02T07:21:29.769 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-09-02T07:21:29.834 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-09-02T07:21:29.971 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-09-02T07:21:30.047 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-09-02T07:21:30.153 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-09-02T07:21:30.239 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-09-02T07:21:30.347 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-09-02T07:21:30.422 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-09-02T07:21:30.535 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-09-02T07:21:30.619 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-09-02T07:21:30.725 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-09-02T07:21:30.812 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-09-02T07:21:30.915 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-09-02T07:21:31.006 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-09-02T07:21:31.132 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-09-02T07:21:31.196 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-09-02T07:21:31.324 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-09-02T07:21:31.389 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-09-02T07:21:31.506 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-09-02T07:21:31.579 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-09-02T07:21:31.706 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-09-02T07:21:31.770 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-09-02T07:21:31.896 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-09-02T07:21:31.959 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-09-02T07:21:32.085 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-09-02T07:21:32.160 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-09-02T07:21:32.294 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-09-02T07:21:32.421 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-09-02T07:21:32.510 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-09-02T07:21:32.655 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-09-02T07:21:32.717 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-09-02T07:21:32.795 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:32 +0000 (0:00:09.393) 0:01:29.305 **** 2021-09-02T07:21:32.891 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-09-02T07:21:33.087 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-09-02T07:21:33.137 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:33 +0000 (0:00:00.342) 0:01:29.647 **** 2021-09-02T07:21:33.381 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-09-02T07:21:33.382 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "checksum": "86e4a2dae5f62bb631669de9d1a6c6efe9b72935", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "5694a3125663295343f0bc40f8059adb", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1630567292.92046-1259-264749888463836/source", "state": "file", "uid": 0} 2021-09-02T07:21:33.403 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:33 +0000 (0:00:00.266) 0:01:29.914 **** 2021-09-02T07:21:33.476 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** ok: [smithi197.front.sepia.ceph.com] => { "msg": "Host smithi197.front.sepia.ceph.com is running CentOS 8.3 (n/a)" } 2021-09-02T07:21:33.498 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:33 +0000 (0:00:00.095) 0:01:30.009 **** 2021-09-02T07:21:33.579 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi197.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-09-02T07:21:33.603 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:33 +0000 (0:00:00.104) 0:01:30.114 **** 2021-09-02T07:21:33.676 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:33.698 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:33 +0000 (0:00:00.094) 0:01:30.208 **** 2021-09-02T07:21:33.706 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "checksum": "86e4a2dae5f62bb631669de9d1a6c6efe9b72935", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "5694a3125663295343f0bc40f8059adb", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1630567293.2350457-1334-118312097359881/source", "state": "file", "uid": 0} 2021-09-02T07:21:33.731 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:33 +0000 (0:00:00.033) 0:01:30.242 **** 2021-09-02T07:21:33.813 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-09-02T07:21:33.815 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => { "msg": "Host smithi175.front.sepia.ceph.com is running CentOS 8.3 (n/a)" } 2021-09-02T07:21:33.837 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:33 +0000 (0:00:00.105) 0:01:30.348 **** 2021-09-02T07:21:33.940 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi175.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-09-02T07:21:33.963 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:33 +0000 (0:00:00.124) 0:01:30.472 **** 2021-09-02T07:21:34.010 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-09-02T07:21:34.012 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.061261", "end": "2021-09-02 07:21:33.983969", "rc": 0, "start": "2021-09-02 07:21:33.922708", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-09-02T07:21:34.034 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:34 +0000 (0:00:00.072) 0:01:30.544 **** 2021-09-02T07:21:34.041 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-09-02T07:21:34.042 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:34.065 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:34 +0000 (0:00:00.031) 0:01:30.575 **** 2021-09-02T07:21:34.103 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-09-02T07:21:34.105 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:34.129 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:34 +0000 (0:00:00.062) 0:01:30.637 **** 2021-09-02T07:21:34.207 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-09-02T07:21:34.209 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:34.234 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:34 +0000 (0:00:00.106) 0:01:30.744 **** 2021-09-02T07:21:34.324 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-09-02T07:21:34.326 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:34.349 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:34 +0000 (0:00:00.115) 0:01:30.859 **** 2021-09-02T07:21:34.387 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-09-02T07:21:34.389 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.074046", "end": "2021-09-02 07:21:34.366717", "rc": 0, "start": "2021-09-02 07:21:34.292671", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-09-02T07:21:34.412 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:34 +0000 (0:00:00.063) 0:01:30.922 **** 2021-09-02T07:21:34.431 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-09-02T07:21:34.432 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:34.455 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:34 +0000 (0:00:00.042) 0:01:30.965 **** 2021-09-02T07:21:34.494 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:34.518 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:34 +0000 (0:00:00.061) 0:01:31.027 **** 2021-09-02T07:21:34.536 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-09-02T07:21:34.538 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:34.560 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:34 +0000 (0:00:00.042) 0:01:31.070 **** 2021-09-02T07:21:34.587 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-09-02T07:21:34.589 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:34.615 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:34 +0000 (0:00:00.055) 0:01:31.125 **** 2021-09-02T07:21:34.639 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:34.664 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:34 +0000 (0:00:00.048) 0:01:31.174 **** 2021-09-02T07:21:34.691 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:34.724 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:34 +0000 (0:00:00.059) 0:01:31.234 **** 2021-09-02T07:21:34.748 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi197.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-02T07:21:34.750 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-02T07:21:34.789 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:34 +0000 (0:00:00.063) 0:01:31.297 **** 2021-09-02T07:21:34.833 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-09-02T07:21:34.835 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:34.860 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:34 +0000 (0:00:00.072) 0:01:31.370 **** 2021-09-02T07:21:34.879 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-09-02T07:21:34.880 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:34.904 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:34 +0000 (0:00:00.044) 0:01:31.414 **** 2021-09-02T07:21:34.933 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:34.957 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:34 +0000 (0:00:00.052) 0:01:31.466 **** 2021-09-02T07:21:34.980 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:35.015 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:35 +0000 (0:00:00.056) 0:01:31.523 **** 2021-09-02T07:21:35.032 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:35.056 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:35 +0000 (0:00:00.042) 0:01:31.566 **** 2021-09-02T07:21:35.099 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:35.125 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:35 +0000 (0:00:00.068) 0:01:31.635 **** 2021-09-02T07:21:35.143 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-09-02T07:21:35.145 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi175.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-02T07:21:35.167 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:35 +0000 (0:00:00.041) 0:01:31.677 **** 2021-09-02T07:21:35.206 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-09-02T07:21:35.208 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-02T07:21:35.229 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:35 +0000 (0:00:00.062) 0:01:31.739 **** 2021-09-02T07:21:35.256 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-09-02T07:21:35.257 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:35.279 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:35 +0000 (0:00:00.050) 0:01:31.790 **** 2021-09-02T07:21:35.338 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:35.376 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:35 +0000 (0:00:00.095) 0:01:31.885 **** 2021-09-02T07:21:35.382 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-09-02T07:21:35.384 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:35.419 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:35 +0000 (0:00:00.035) 0:01:31.921 **** 2021-09-02T07:21:35.449 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi197.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-02T07:21:35.510 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:35 +0000 (0:00:00.091) 0:01:32.013 **** TASK [common : Set the releasever] ********************************************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:35.539 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:35 +0000 (0:00:00.032) 0:01:32.045 **** 2021-09-02T07:21:35.578 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-09-02T07:21:35.581 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:35.612 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:35 +0000 (0:00:00.076) 0:01:32.122 **** 2021-09-02T07:21:35.623 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-09-02T07:21:35.625 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-02T07:21:35.661 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:35 +0000 (0:00:00.048) 0:01:32.171 **** 2021-09-02T07:21:35.682 INFO:teuthology.task.ansible.out: 2021-09-02T07:21:35.684 INFO:teuthology.task.ansible.out:TASK [common : List CDN releases available to system] ************************** 2021-09-02T07:21:35.686 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:35.710 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:35 +0000 (0:00:00.048) 0:01:32.219 **** 2021-09-02T07:21:35.751 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:35.774 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:35 +0000 (0:00:00.060) 0:01:32.279 **** 2021-09-02T07:21:35.807 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:35.834 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:35 +0000 (0:00:00.062) 0:01:32.341 **** 2021-09-02T07:21:35.867 INFO:teuthology.task.ansible.out: 2021-09-02T07:21:35.869 INFO:teuthology.task.ansible.out:TASK [common : Register with subscription-manager.] **************************** skipping: [smithi175.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-02T07:21:35.893 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:35 +0000 (0:00:00.060) 0:01:32.402 **** 2021-09-02T07:21:35.926 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:35.951 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:35 +0000 (0:00:00.058) 0:01:32.461 **** 2021-09-02T07:21:35.955 INFO:teuthology.task.ansible.out: 2021-09-02T07:21:35.971 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-09-02T07:21:35.973 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:35.998 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:35 +0000 (0:00:00.047) 0:01:32.508 **** 2021-09-02T07:21:36.016 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-09-02T07:21:36.018 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-02T07:21:36.057 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:36 +0000 (0:00:00.046) 0:01:32.555 **** 2021-09-02T07:21:36.073 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-09-02T07:21:36.075 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:36.097 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:36 +0000 (0:00:00.052) 0:01:32.607 **** 2021-09-02T07:21:36.126 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:36.151 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:36 +0000 (0:00:00.052) 0:01:32.660 **** 2021-09-02T07:21:36.195 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:36.219 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:36 +0000 (0:00:00.067) 0:01:32.728 **** 2021-09-02T07:21:36.236 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:36.267 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:36 +0000 (0:00:00.048) 0:01:32.777 **** 2021-09-02T07:21:36.301 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:36.322 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:36 +0000 (0:00:00.055) 0:01:32.832 **** 2021-09-02T07:21:36.351 INFO:teuthology.task.ansible.out: 2021-09-02T07:21:36.352 INFO:teuthology.task.ansible.out:TASK [common : Store list of enabled repos] ************************************ skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:36.382 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:36 +0000 (0:00:00.057) 0:01:32.889 **** 2021-09-02T07:21:36.409 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi175.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-02T07:21:36.437 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:36 +0000 (0:00:00.056) 0:01:32.946 **** 2021-09-02T07:21:36.473 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:36.504 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:36 +0000 (0:00:00.068) 0:01:33.014 **** 2021-09-02T07:21:36.520 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-09-02T07:21:36.522 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:36.550 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:36 +0000 (0:00:00.045) 0:01:33.059 **** 2021-09-02T07:21:36.606 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-09-02T07:21:36.607 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:36.630 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:36 +0000 (0:00:00.080) 0:01:33.140 **** 2021-09-02T07:21:36.646 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-09-02T07:21:36.648 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:36.672 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:36 +0000 (0:00:00.041) 0:01:33.182 **** 2021-09-02T07:21:36.722 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:36.758 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:36 +0000 (0:00:00.075) 0:01:33.258 **** 2021-09-02T07:21:36.781 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:36.807 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:36 +0000 (0:00:00.058) 0:01:33.317 **** 2021-09-02T07:21:36.854 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi197.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-02T07:21:36.878 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:36 +0000 (0:00:00.071) 0:01:33.388 **** 2021-09-02T07:21:36.897 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-09-02T07:21:36.899 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:36.925 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:36 +0000 (0:00:00.044) 0:01:33.433 **** 2021-09-02T07:21:36.963 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:36.988 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:36 +0000 (0:00:00.063) 0:01:33.497 **** 2021-09-02T07:21:37.005 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-09-02T07:21:37.012 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:37.030 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:37 +0000 (0:00:00.043) 0:01:33.540 **** 2021-09-02T07:21:37.062 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-09-02T07:21:37.066 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:37.087 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:37 +0000 (0:00:00.056) 0:01:33.597 **** 2021-09-02T07:21:37.106 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:37.130 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:37 +0000 (0:00:00.043) 0:01:33.640 **** 2021-09-02T07:21:37.256 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi175.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-02T07:21:37.274 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:37 +0000 (0:00:00.138) 0:01:33.779 **** 2021-09-02T07:21:37.332 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:37.357 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:37 +0000 (0:00:00.088) 0:01:33.867 **** 2021-09-02T07:21:37.365 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-09-02T07:21:37.367 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-02T07:21:37.390 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:37 +0000 (0:00:00.032) 0:01:33.900 **** 2021-09-02T07:21:37.442 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:37.471 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:37 +0000 (0:00:00.076) 0:01:33.976 **** 2021-09-02T07:21:37.756 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-09-02T07:21:37.757 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-02T07:21:37.782 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:37 +0000 (0:00:00.315) 0:01:34.292 **** 2021-09-02T07:21:38.237 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** ok: [smithi197.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-09-02T07:21:38.345 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-09-02T07:21:38.670 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-09-02T07:21:38.700 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:38 +0000 (0:00:00.918) 0:01:35.211 **** 2021-09-02T07:21:38.808 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-09-02T07:21:38.836 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:38 +0000 (0:00:00.135) 0:01:35.346 **** 2021-09-02T07:21:39.242 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-09-02T07:21:39.245 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-09-02T07:21:39.373 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-09-02T07:21:39.681 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-09-02T07:21:39.716 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:39 +0000 (0:00:00.874) 0:01:36.220 **** 2021-09-02T07:21:39.784 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-09-02T07:21:39.786 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:39.808 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:39 +0000 (0:00:00.097) 0:01:36.318 **** 2021-09-02T07:21:39.828 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* ok: [smithi175.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-09-02T07:21:39.856 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:39 +0000 (0:00:00.048) 0:01:36.366 **** 2021-09-02T07:21:39.890 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-09-02T07:21:39.891 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:39.932 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:39 +0000 (0:00:00.075) 0:01:36.442 **** 2021-09-02T07:21:39.949 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-09-02T07:21:39.951 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:39.976 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:39 +0000 (0:00:00.043) 0:01:36.486 **** 2021-09-02T07:21:40.016 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:40.048 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:40 +0000 (0:00:00.064) 0:01:36.550 **** 2021-09-02T07:21:40.079 INFO:teuthology.task.ansible.out: 2021-09-02T07:21:40.080 INFO:teuthology.task.ansible.out:TASK [common : Get the current timezone] *************************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:40.110 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:40 +0000 (0:00:00.068) 0:01:36.619 **** 2021-09-02T07:21:40.151 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-09-02T07:21:40.153 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:40.175 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:40 +0000 (0:00:00.065) 0:01:36.685 **** 2021-09-02T07:21:40.195 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-09-02T07:21:40.197 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:40.220 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:40 +0000 (0:00:00.044) 0:01:36.730 **** 2021-09-02T07:21:40.258 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** skipping: [smithi197.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-09-02T07:21:40.272 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-09-02T07:21:40.283 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-09-02T07:21:40.312 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:40 +0000 (0:00:00.091) 0:01:36.821 **** 2021-09-02T07:21:40.321 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-09-02T07:21:40.323 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:40.351 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:40 +0000 (0:00:00.039) 0:01:36.861 **** 2021-09-02T07:21:40.405 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:40.432 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:40 +0000 (0:00:00.080) 0:01:36.941 **** 2021-09-02T07:21:40.446 INFO:teuthology.task.ansible.out: 2021-09-02T07:21:40.450 INFO:teuthology.task.ansible.out:TASK [common : Mask sleep units] *********************************************** skipping: [smithi175.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-09-02T07:21:40.480 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:40 +0000 (0:00:00.045) 0:01:36.987 **** 2021-09-02T07:21:40.532 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:40.556 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:40 +0000 (0:00:00.078) 0:01:37.065 **** 2021-09-02T07:21:40.567 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:40.605 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:40 +0000 (0:00:00.050) 0:01:37.116 **** 2021-09-02T07:21:40.607 INFO:teuthology.task.ansible.out: 2021-09-02T07:21:40.629 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-09-02T07:21:40.631 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:40.662 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:40 +0000 (0:00:00.051) 0:01:37.167 **** 2021-09-02T07:21:40.688 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:40.724 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:40 +0000 (0:00:00.066) 0:01:37.234 **** 2021-09-02T07:21:40.751 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:40.786 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:40 +0000 (0:00:00.058) 0:01:37.293 **** 2021-09-02T07:21:40.820 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:40.846 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:40 +0000 (0:00:00.061) 0:01:37.354 **** 2021-09-02T07:21:40.863 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:40.897 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:40 +0000 (0:00:00.049) 0:01:37.404 **** 2021-09-02T07:21:40.922 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-09-02T07:21:40.924 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:40.947 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:40 +0000 (0:00:00.053) 0:01:37.457 **** 2021-09-02T07:21:41.020 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:41.042 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:41 +0000 (0:00:00.094) 0:01:37.552 **** 2021-09-02T07:21:41.525 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.278928", "end": "2021-09-02 07:21:41.500336", "rc": 0, "start": "2021-09-02 07:21:41.221408", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.432790", "end": "2021-09-02 07:21:41.500342", "rc": 0, "start": "2021-09-02 07:21:41.067552", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:21:41.547 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:41 +0000 (0:00:00.501) 0:01:38.053 **** 2021-09-02T07:21:41.572 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:41 +0000 (0:00:00.028) 0:01:38.082 **** 2021-09-02T07:21:45.575 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-02T07:21:45.599 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:45 +0000 (0:00:04.023) 0:01:42.106 **** 2021-09-02T07:21:45.648 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-02T07:21:45.670 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:45 +0000 (0:00:00.074) 0:01:42.180 **** 2021-09-02T07:21:45.681 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:45.702 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:45 +0000 (0:00:00.030) 0:01:42.211 **** 2021-09-02T07:21:45.742 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:45.775 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:45 +0000 (0:00:00.067) 0:01:42.278 **** 2021-09-02T07:21:45.789 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:45.818 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:45 +0000 (0:00:00.046) 0:01:42.324 **** 2021-09-02T07:21:45.855 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:45.881 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:45 +0000 (0:00:00.065) 0:01:42.390 **** 2021-09-02T07:21:45.911 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:45.935 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:45 +0000 (0:00:00.055) 0:01:42.445 **** 2021-09-02T07:21:45.963 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:45.991 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:45 +0000 (0:00:00.056) 0:01:42.502 **** 2021-09-02T07:21:46.012 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:46.037 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:46 +0000 (0:00:00.045) 0:01:42.547 **** 2021-09-02T07:21:46.067 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:46.092 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:46 +0000 (0:00:00.054) 0:01:42.602 **** 2021-09-02T07:21:46.542 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-09-02T07:21:46.565 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:46 +0000 (0:00:00.473) 0:01:43.075 **** 2021-09-02T07:21:46.651 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-09-02T07:21:46.688 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:46 +0000 (0:00:00.119) 0:01:43.195 **** 2021-09-02T07:21:48.173 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-09-02T07:21:48.175 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-02T07:21:48.196 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:48 +0000 (0:00:01.512) 0:01:44.707 **** 2021-09-02T07:21:48.258 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-02T07:21:48.286 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:48 +0000 (0:00:00.089) 0:01:44.796 **** 2021-09-02T07:21:48.741 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-09-02T07:21:48.743 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-09-02T07:21:48.832 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-09-02T07:21:49.186 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-09-02T07:21:49.270 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-09-02T07:21:49.613 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-09-02T07:21:49.637 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:49 +0000 (0:00:01.351) 0:01:46.148 **** 2021-09-02T07:21:49.710 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-09-02T07:21:49.743 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:49 +0000 (0:00:00.105) 0:01:46.254 **** 2021-09-02T07:21:49.827 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-02T07:21:49.853 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:49 +0000 (0:00:00.109) 0:01:46.363 **** 2021-09-02T07:21:49.938 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-02T07:21:49.963 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:49 +0000 (0:00:00.108) 0:01:46.473 **** 2021-09-02T07:21:50.379 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-09-02T07:21:50.381 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-09-02T07:21:50.496 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-09-02T07:21:50.795 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-09-02T07:21:50.926 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-09-02T07:21:51.213 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "state": "file", "uid": 0} 2021-09-02T07:21:51.238 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:51 +0000 (0:00:01.274) 0:01:47.748 **** 2021-09-02T07:21:51.308 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-09-02T07:21:51.309 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:51.332 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:51 +0000 (0:00:00.092) 0:01:47.840 **** 2021-09-02T07:21:51.349 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi197.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "state": "file", "uid": 0} 2021-09-02T07:21:51.374 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:51 +0000 (0:00:00.043) 0:01:47.884 **** 2021-09-02T07:21:51.414 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** ok: [smithi175.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-09-02T07:21:51.437 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:51 +0000 (0:00:00.062) 0:01:47.947 **** 2021-09-02T07:21:51.453 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-09-02T07:21:51.455 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:51.476 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:51 +0000 (0:00:00.039) 0:01:47.986 **** 2021-09-02T07:21:51.550 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** ok: [smithi197.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-09-02T07:21:51.572 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:51 +0000 (0:00:00.095) 0:01:48.082 **** 2021-09-02T07:21:51.821 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-09-02T07:21:51.823 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-02T07:21:51.853 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:51 +0000 (0:00:00.280) 0:01:48.363 **** 2021-09-02T07:21:51.947 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-09-02T07:21:51.948 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-02T07:21:51.958 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-02T07:21:51.980 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-02T07:21:51.991 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-02T07:21:52.017 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:52 +0000 (0:00:00.163) 0:01:48.526 **** 2021-09-02T07:21:52.679 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-09-02T07:21:52.681 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-02T07:21:52.701 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:52 +0000 (0:00:00.685) 0:01:49.211 **** 2021-09-02T07:21:52.793 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-09-02T07:21:52.795 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-02T07:21:52.804 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-02T07:21:52.815 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-02T07:21:52.837 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-02T07:21:52.861 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:52 +0000 (0:00:00.159) 0:01:49.371 **** 2021-09-02T07:21:53.632 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-09-02T07:21:53.634 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-02T07:21:53.666 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:53 +0000 (0:00:00.805) 0:01:50.177 **** 2021-09-02T07:21:53.739 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-09-02T07:21:53.741 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:53.762 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:53 +0000 (0:00:00.095) 0:01:50.272 **** 2021-09-02T07:21:54.266 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-09-02T07:21:54.268 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-09-02T07:21:54.288 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:54 +0000 (0:00:00.526) 0:01:50.799 **** 2021-09-02T07:21:54.494 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-09-02T07:21:54.497 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-02T07:21:54.519 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:54 +0000 (0:00:00.230) 0:01:51.029 **** 2021-09-02T07:21:54.588 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-09-02T07:21:54.590 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:54.613 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:54 +0000 (0:00:00.094) 0:01:51.123 **** 2021-09-02T07:21:54.799 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-09-02T07:21:54.801 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-09-02T07:21:54.822 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:54 +0000 (0:00:00.208) 0:01:51.332 **** 2021-09-02T07:21:54.902 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ skipping: [smithi197.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-09-02T07:21:54.923 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-09-02T07:21:54.949 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:54 +0000 (0:00:00.127) 0:01:51.460 **** 2021-09-02T07:21:55.116 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-09-02T07:21:55.117 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-09-02T07:21:55.140 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:55 +0000 (0:00:00.190) 0:01:51.650 **** 2021-09-02T07:21:55.189 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-09-02T07:21:55.191 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-02T07:21:55.213 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:55 +0000 (0:00:00.072) 0:01:51.723 **** 2021-09-02T07:21:55.497 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** fatal: [smithi197.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-09-02T07:21:55.523 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:55 +0000 (0:00:00.309) 0:01:52.033 **** 2021-09-02T07:21:55.591 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-09-02T07:21:55.593 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:55.614 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:55 +0000 (0:00:00.091) 0:01:52.124 **** 2021-09-02T07:21:55.652 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-09-02T07:21:55.654 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-09-02T07:21:55.677 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:55 +0000 (0:00:00.062) 0:01:52.187 **** 2021-09-02T07:21:55.779 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-09-02T07:21:55.781 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-09-02T07:21:55.790 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-09-02T07:21:55.812 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:55 +0000 (0:00:00.135) 0:01:52.322 **** 2021-09-02T07:21:56.054 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* ok: [smithi175.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-02T07:21:56.082 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:56 +0000 (0:00:00.267) 0:01:52.590 **** 2021-09-02T07:21:56.119 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-09-02T07:21:56.124 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-09-02T07:21:56.144 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:56 +0000 (0:00:00.064) 0:01:52.654 **** 2021-09-02T07:21:56.362 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-09-02T07:21:56.364 INFO:teuthology.task.ansible.out:fatal: [smithi175.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-09-02T07:21:56.385 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:56 +0000 (0:00:00.241) 0:01:52.895 **** 2021-09-02T07:21:56.465 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:21:56.488 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:56 +0000 (0:00:00.103) 0:01:52.999 **** 2021-09-02T07:21:56.820 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-09-02T07:21:56.822 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveState": "active", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1296", "ExecMainStartTimestamp": "Thu 2021-09-02 07:19:56 UTC", "ExecMainStartTimestampMonotonic": "19636299", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Thu 2021-09-02 07:19:56 UTC] ; stop_time=[n/a] ; pid=1296 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "984", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "L 2021-09-02T07:21:56.823 INFO:teuthology.task.ansible.out:imitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127246", "LimitNPROCSoft": "127246", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127246", "LimitSIGPENDINGSoft": "127246", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "1296", "MemoryAccounting": "yes", "MemoryCurrent": "671744", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "on-abort", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203593", "TimeoutStar 2021-09-02T07:21:56.825 INFO:teuthology.task.ansible.out:tUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Type": "simple", "UID": "988", "UMask": "0022", "WatchdogTimestamp": "Thu 2021-09-02 07:19:56 UTC", "WatchdogTimestampMonotonic": "19636479", "WatchdogUSec": "0"}} 2021-09-02T07:21:56.846 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:56 +0000 (0:00:00.357) 0:01:53.356 **** 2021-09-02T07:21:56.994 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-09-02T07:21:57.018 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:57 +0000 (0:00:00.172) 0:01:53.529 **** 2021-09-02T07:21:57.078 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-09-02T07:21:57.080 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001724", "end": "2021-09-02 07:21:57.053843", "rc": 0, "start": "2021-09-02 07:21:57.052119", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-09-02T07:21:57.103 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:57 +0000 (0:00:00.084) 0:01:53.613 **** 2021-09-02T07:21:57.371 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-09-02T07:21:57.373 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveState": "active", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1253", "ExecMainStartTimestamp": "Thu 2021-09-02 07:19:53 UTC", "ExecMainStartTimestampMonotonic": "20026643", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Thu 2021-09-02 07:19:53 UTC] ; stop_time=[n/a] ; pid=1253 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "984", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "L 2021-09-02T07:21:57.375 INFO:teuthology.task.ansible.out:imitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "1253", "MemoryAccounting": "yes", "MemoryCurrent": "675840", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "on-abort", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203592", "TimeoutStar 2021-09-02T07:21:57.378 INFO:teuthology.task.ansible.out:tUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Type": "simple", "UID": "988", "UMask": "0022", "WatchdogTimestamp": "Thu 2021-09-02 07:19:53 UTC", "WatchdogTimestampMonotonic": "20026687", "WatchdogUSec": "0"}} 2021-09-02T07:21:57.403 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:57 +0000 (0:00:00.299) 0:01:53.913 **** 2021-09-02T07:21:57.623 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-09-02T07:21:57.625 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001812", "end": "2021-09-02 07:21:57.606355", "rc": 0, "start": "2021-09-02 07:21:57.604543", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-09-02T07:21:57.645 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:57 +0000 (0:00:00.242) 0:01:54.156 **** 2021-09-02T07:21:58.712 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-09-02T07:21:58.714 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-02T07:21:58.736 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:58 +0000 (0:00:01.090) 0:01:55.246 **** 2021-09-02T07:21:59.230 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-02T07:21:59.253 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:21:59 +0000 (0:00:00.516) 0:01:55.762 **** 2021-09-02T07:22:36.331 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-09-02T07:22:36.333 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted-3.14.3-67.el8_4.1.noarch", "Installed: selinux-policy-3.14.3-67.el8_4.1.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch"]} 2021-09-02T07:22:36.355 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:22:36 +0000 (0:00:37.102) 0:02:32.865 **** 2021-09-02T07:22:36.457 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted-3.14.3-67.el8_4.1.noarch", "Installed: selinux-policy-3.14.3-67.el8_4.1.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch"]} 2021-09-02T07:22:36.482 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:22:36 +0000 (0:00:00.127) 0:02:32.992 **** 2021-09-02T07:22:37.090 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-09-02T07:22:37.092 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-09-02T07:22:37.114 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:22:37 +0000 (0:00:00.631) 0:02:33.624 **** 2021-09-02T07:22:37.135 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:22:37 +0000 (0:00:00.020) 0:02:33.644 **** 2021-09-02T07:22:52.091 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:14.733409", "end": "2021-09-02 07:22:52.067810", "failed_when_result": false, "rc": 0, "start": "2021-09-02 07:22:37.334401", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-09-02T07:22:52.114 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:22:52 +0000 (0:00:14.980) 0:02:48.624 **** 2021-09-02T07:22:52.178 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:14.797018", "end": "2021-09-02 07:22:52.148720", "failed_when_result": false, "rc": 0, "start": "2021-09-02 07:22:37.351702", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-09-02T07:22:52.205 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:22:52 +0000 (0:00:00.090) 0:02:48.715 **** 2021-09-02T07:22:52.632 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-09-02T07:22:52.655 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:22:52 +0000 (0:00:00.450) 0:02:49.165 **** 2021-09-02T07:22:52.725 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-09-02T07:22:52.756 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:22:52 +0000 (0:00:00.100) 0:02:49.266 **** 2021-09-02T07:22:52.897 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-09-02T07:22:52.899 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.011201", "end": "2021-09-02 07:22:52.871920", "rc": 0, "start": "2021-09-02 07:22:52.860719", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:22:52.923 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:22:52 +0000 (0:00:00.166) 0:02:49.433 **** 2021-09-02T07:22:53.019 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.012116", "end": "2021-09-02 07:22:52.994080", "rc": 0, "start": "2021-09-02 07:22:52.981964", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:22:53.044 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:22:53 +0000 (0:00:00.120) 0:02:49.554 **** 2021-09-02T07:22:53.166 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-09-02T07:22:53.168 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.006452", "end": "2021-09-02 07:22:53.142650", "rc": 0, "start": "2021-09-02 07:22:53.136198", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:22:53.190 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:22:53 +0000 (0:00:00.146) 0:02:49.701 **** 2021-09-02T07:22:53.284 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.010763", "end": "2021-09-02 07:22:53.259428", "rc": 0, "start": "2021-09-02 07:22:53.248665", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:22:53.311 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:22:53 +0000 (0:00:00.120) 0:02:49.821 **** 2021-09-02T07:23:08.241 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:14.809263", "end": "2021-09-02 07:23:08.214831", "rc": 0, "start": "2021-09-02 07:22:53.405568", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:23:08.258 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:08 +0000 (0:00:14.947) 0:03:04.769 **** 2021-09-02T07:23:08.382 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:14.843134", "end": "2021-09-02 07:23:08.360514", "rc": 0, "start": "2021-09-02 07:22:53.517380", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:23:08.405 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:08 +0000 (0:00:00.146) 0:03:04.915 **** 2021-09-02T07:23:08.518 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-09-02T07:23:08.520 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-09-02T07:23:08.544 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:08 +0000 (0:00:00.139) 0:03:05.054 **** 2021-09-02T07:23:08.626 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-09-02T07:23:08.628 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-09-02T07:23:08.650 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:08 +0000 (0:00:00.105) 0:03:05.160 **** 2021-09-02T07:23:08.658 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-09-02T07:23:08.661 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-09-02T07:23:08.683 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:08 +0000 (0:00:00.033) 0:03:05.193 **** 2021-09-02T07:23:08.723 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-09-02T07:23:08.724 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-02T07:23:08.750 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:08 +0000 (0:00:00.066) 0:03:05.260 **** 2021-09-02T07:23:08.768 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi197.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-09-02T07:23:08.792 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:08 +0000 (0:00:00.041) 0:03:05.302 **** 2021-09-02T07:23:08.841 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** ok: [smithi175.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.rep 2021-09-02T07:23:08.843 INFO:teuthology.task.ansible.out:os.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-09-02T07:23:08.864 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:08 +0000 (0:00:00.072) 0:03:05.375 **** 2021-09-02T07:23:08.883 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-09-02T07:23:08.885 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-02T07:23:08.907 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:08 +0000 (0:00:00.042) 0:03:05.417 **** 2021-09-02T07:23:08.946 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi175.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-02T07:23:08.971 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:08 +0000 (0:00:00.063) 0:03:05.481 **** 2021-09-02T07:23:08.992 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** ok: [smithi197.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.rep 2021-09-02T07:23:08.995 INFO:teuthology.task.ansible.out:os.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-09-02T07:23:09.018 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:09 +0000 (0:00:00.046) 0:03:05.528 **** 2021-09-02T07:23:09.088 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-09-02T07:23:09.089 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-02T07:23:09.111 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:09 +0000 (0:00:00.092) 0:03:05.621 **** 2021-09-02T07:23:09.171 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-09-02T07:23:09.173 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-09-02T07:23:09.195 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:09 +0000 (0:00:00.084) 0:03:05.706 **** 2021-09-02T07:23:09.296 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-09-02T07:23:09.322 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:09 +0000 (0:00:00.126) 0:03:05.832 **** 2021-09-02T07:23:09.434 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-09-02T07:23:09.436 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "gid": 1254, "name": "ubuntu", "state": "present", "system": false} 2021-09-02T07:23:09.457 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:09 +0000 (0:00:00.135) 0:03:05.967 **** 2021-09-02T07:23:09.527 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-09-02T07:23:09.529 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:09.549 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:09 +0000 (0:00:00.091) 0:03:06.059 **** 2021-09-02T07:23:09.566 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-09-02T07:23:09.568 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "gid": 1254, "name": "ubuntu", "state": "present", "system": false} 2021-09-02T07:23:09.590 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:09 +0000 (0:00:00.040) 0:03:06.100 **** 2021-09-02T07:23:09.619 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-09-02T07:23:09.621 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:09.645 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:09 +0000 (0:00:00.053) 0:03:06.154 **** 2021-09-02T07:23:09.661 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-09-02T07:23:09.663 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:09.687 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:09 +0000 (0:00:00.042) 0:03:06.196 **** 2021-09-02T07:23:09.758 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:09.782 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:09 +0000 (0:00:00.095) 0:03:06.292 **** 2021-09-02T07:23:09.914 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-09-02T07:23:09.915 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-09-02T07:23:09.940 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:09 +0000 (0:00:00.158) 0:03:06.450 **** 2021-09-02T07:23:10.051 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-09-02T07:23:10.074 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:10 +0000 (0:00:00.134) 0:03:06.585 **** 2021-09-02T07:23:10.516 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-09-02T07:23:10.518 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-09-02T07:23:10.539 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:10 +0000 (0:00:00.464) 0:03:07.049 **** 2021-09-02T07:23:10.608 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-09-02T07:23:10.609 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi175"}, "changed": false} 2021-09-02T07:23:10.632 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:10 +0000 (0:00:00.092) 0:03:07.142 **** 2021-09-02T07:23:10.653 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-09-02T07:23:10.654 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-09-02T07:23:10.684 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:10 +0000 (0:00:00.052) 0:03:07.195 **** 2021-09-02T07:23:10.771 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi197.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi197"}, "changed": false} 2021-09-02T07:23:10.794 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:10 +0000 (0:00:00.109) 0:03:07.304 **** 2021-09-02T07:23:11.568 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ changed: [smithi197.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi197.front.sepia.ceph.com", "ansible_hostname": "smithi197", "ansible_nodename": "smithi197"}, "changed": true, "name": "smithi197"} 2021-09-02T07:23:11.590 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:11 +0000 (0:00:00.795) 0:03:08.100 **** 2021-09-02T07:23:11.592 INFO:teuthology.task.ansible.out: 2021-09-02T07:23:11.608 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi175.front.sepia.ceph.com", "ansible_hostname": "smithi175", "ansible_nodename": "smithi175"}, "changed": true, "name": "smithi175"} 2021-09-02T07:23:11.631 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:11 +0000 (0:00:00.040) 0:03:08.141 **** 2021-09-02T07:23:12.077 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-09-02T07:23:12.100 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:12 +0000 (0:00:00.469) 0:03:08.610 **** 2021-09-02T07:23:12.119 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-09-02T07:23:12.147 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:12 +0000 (0:00:00.046) 0:03:08.657 **** 2021-09-02T07:23:12.606 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-09-02T07:23:12.608 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-09-02T07:23:12.629 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:12 +0000 (0:00:00.481) 0:03:09.139 **** 2021-09-02T07:23:12.646 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-09-02T07:23:12.669 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:12 +0000 (0:00:00.040) 0:03:09.180 **** 2021-09-02T07:23:13.204 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** changed: [smithi197.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-09-02T07:23:13.229 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:13 +0000 (0:00:00.559) 0:03:09.739 **** 2021-09-02T07:23:13.262 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-09-02T07:23:13.289 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:13 +0000 (0:00:00.059) 0:03:09.799 **** 2021-09-02T07:23:13.737 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-09-02T07:23:13.761 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:13 +0000 (0:00:00.472) 0:03:10.271 **** 2021-09-02T07:23:13.802 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-09-02T07:23:13.836 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:13 +0000 (0:00:00.074) 0:03:10.346 **** 2021-09-02T07:23:14.003 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-09-02T07:23:14.005 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-09-02T07:23:14.027 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:14 +0000 (0:00:00.191) 0:03:10.537 **** 2021-09-02T07:23:14.087 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-09-02T07:23:14.110 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:14 +0000 (0:00:00.082) 0:03:10.620 **** 2021-09-02T07:23:14.265 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-09-02T07:23:14.266 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-09-02T07:23:14.288 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:14 +0000 (0:00:00.178) 0:03:10.799 **** 2021-09-02T07:23:14.350 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-09-02T07:23:14.377 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:14 +0000 (0:00:00.088) 0:03:10.887 **** 2021-09-02T07:23:14.554 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* ok: [smithi197.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1254, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-09-02T07:23:14.585 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:14 +0000 (0:00:00.207) 0:03:11.095 **** 2021-09-02T07:23:14.666 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1254, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-09-02T07:23:14.690 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:14 +0000 (0:00:00.104) 0:03:11.200 **** 2021-09-02T07:23:15.089 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-09-02T07:23:15.091 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-09-02T07:23:15.114 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:15 +0000 (0:00:00.423) 0:03:11.624 **** 2021-09-02T07:23:15.196 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-09-02T07:23:15.219 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:15 +0000 (0:00:00.105) 0:03:11.729 **** 2021-09-02T07:23:15.632 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-09-02T07:23:15.635 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-09-02T07:23:15.657 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:15 +0000 (0:00:00.436) 0:03:12.165 **** 2021-09-02T07:23:15.725 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-09-02T07:23:15.749 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:15 +0000 (0:00:00.093) 0:03:12.259 **** 2021-09-02T07:23:18.211 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-09-02T07:23:18.213 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.322991", "end": "2021-09-02 07:23:18.185069", "rc": 0, "start": "2021-09-02 07:23:15.862078", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:23:18.235 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:18 +0000 (0:00:02.485) 0:03:14.745 **** 2021-09-02T07:23:18.306 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.313059", "end": "2021-09-02 07:23:18.272635", "rc": 0, "start": "2021-09-02 07:23:15.959576", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:23:18.323 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:18 +0000 (0:00:00.088) 0:03:14.834 **** 2021-09-02T07:23:18.331 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:18.358 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:18 +0000 (0:00:00.031) 0:03:14.865 **** 2021-09-02T07:23:18.395 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:18.421 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:18 +0000 (0:00:00.065) 0:03:14.930 **** 2021-09-02T07:23:18.430 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-02T07:23:18.432 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:18.455 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:18 +0000 (0:00:00.034) 0:03:14.965 **** 2021-09-02T07:23:18.488 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:18.510 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:18 +0000 (0:00:00.055) 0:03:15.020 **** 2021-09-02T07:23:18.539 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2021-09-02T07:23:18.541 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml", "skip_reason": "Conditional result was False"} 2021-09-02T07:23:18.576 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:18 +0000 (0:00:00.053) 0:03:15.074 **** 2021-09-02T07:23:18.593 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml", "skip_reason": "Conditional result was False"} 2021-09-02T07:23:18.627 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:18 +0000 (0:00:00.058) 0:03:15.132 **** 2021-09-02T07:23:19.102 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-09-02T07:23:19.104 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-09-02T07:23:19.146 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-09-02T07:23:19.552 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2021-09-02T07:23:19.574 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2021-09-02T07:23:19.976 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2021-09-02T07:23:20.008 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2021-09-02T07:23:20.417 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=CentOS-PowerTools-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "20107eac0ba9a84b318fa5f29b02a387051bff98", "dest": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-PowerTools-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-09-02T07:23:20.453 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:20 +0000 (0:00:01.830) 0:03:16.963 **** 2021-09-02T07:23:20.463 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=CentOS-PowerTools-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "20107eac0ba9a84b318fa5f29b02a387051bff98", "dest": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-PowerTools-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-09-02T07:23:20.498 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:20 +0000 (0:00:00.044) 0:03:17.008 **** 2021-09-02T07:23:20.993 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-09-02T07:23:20.995 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-09-02T07:23:21.017 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:21 +0000 (0:00:00.519) 0:03:17.527 **** 2021-09-02T07:23:21.058 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-09-02T07:23:21.083 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:21 +0000 (0:00:00.065) 0:03:17.593 **** 2021-09-02T07:23:21.544 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-09-02T07:23:21.546 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-09-02T07:23:21.599 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-09-02T07:23:21.970 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-09-02T07:23:22.035 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-09-02T07:23:22.416 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-09-02T07:23:22.461 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-09-02T07:23:22.844 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'key': 'CentOS-PowerTools', 'value': {'name': 'CentOS-$releasever - PowerTools', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "76441d1028936932e965a46c12ad393f767201d3", "dest": "/etc/yum.repos.d/CentOS-PowerTools.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-PowerTools", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-PowerTools.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 292, "state": "file", "uid": 0} 2021-09-02T07:23:22.883 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:22 +0000 (0:00:01.798) 0:03:19.391 **** 2021-09-02T07:23:22.900 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item={'key': 'CentOS-PowerTools', 'value': {'name': 'CentOS-$releasever - PowerTools', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "76441d1028936932e965a46c12ad393f767201d3", "dest": "/etc/yum.repos.d/CentOS-PowerTools.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-PowerTools", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-PowerTools.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 292, "state": "file", "uid": 0} 2021-09-02T07:23:22.932 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:22 +0000 (0:00:00.050) 0:03:19.442 **** 2021-09-02T07:23:23.428 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** changed: [smithi197.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.300448", "end": "2021-09-02 07:23:23.400273", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-09-02 07:23:23.099825", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2021-09-02T07:23:23.429 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2021-09-02T07:23:23.454 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:23 +0000 (0:00:00.520) 0:03:19.963 **** 2021-09-02T07:23:23.467 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.278228", "end": "2021-09-02 07:23:23.425397", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-09-02 07:23:23.147169", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2021-09-02T07:23:23.494 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:23 +0000 (0:00:00.039) 0:03:20.003 **** 2021-09-02T07:23:23.530 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-09-02T07:23:23.532 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:23.556 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:23 +0000 (0:00:00.062) 0:03:20.065 **** 2021-09-02T07:23:23.577 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:23.603 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:23 +0000 (0:00:00.046) 0:03:20.112 **** 2021-09-02T07:23:23.628 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-09-02T07:23:23.632 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-09-02T07:23:23.640 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-09-02T07:23:23.664 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:23 +0000 (0:00:00.061) 0:03:20.174 **** 2021-09-02T07:23:23.682 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-09-02T07:23:23.684 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-09-02T07:23:23.709 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:23 +0000 (0:00:00.043) 0:03:20.218 **** 2021-09-02T07:23:23.738 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:23.763 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:23 +0000 (0:00:00.055) 0:03:20.273 **** 2021-09-02T07:23:23.781 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:23.805 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:23 +0000 (0:00:00.041) 0:03:20.315 **** 2021-09-02T07:23:23.835 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi197.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-09-02T07:23:23.859 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:23 +0000 (0:00:00.054) 0:03:20.369 **** 2021-09-02T07:23:23.879 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-09-02T07:23:23.905 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:23 +0000 (0:00:00.043) 0:03:20.413 **** 2021-09-02T07:23:23.934 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:23.962 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:23 +0000 (0:00:00.056) 0:03:20.470 **** 2021-09-02T07:23:23.980 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:24.004 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:23 +0000 (0:00:00.043) 0:03:20.514 **** 2021-09-02T07:23:24.032 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:24.056 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:24 +0000 (0:00:00.052) 0:03:20.567 **** 2021-09-02T07:23:24.074 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:24.097 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:24 +0000 (0:00:00.040) 0:03:20.608 **** 2021-09-02T07:23:24.300 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-09-02T07:23:24.302 INFO:teuthology.task.ansible.out:fatal: [smithi197.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006457", "end": "2021-09-02 07:23:24.276203", "msg": "non-zero return code", "rc": 1, "start": "2021-09-02 07:23:24.269746", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-09-02T07:23:24.324 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:24 +0000 (0:00:00.226) 0:03:20.834 **** 2021-09-02T07:23:24.343 INFO:teuthology.task.ansible.out:fatal: [smithi175.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006697", "end": "2021-09-02 07:23:24.322763", "msg": "non-zero return code", "rc": 1, "start": "2021-09-02 07:23:24.316066", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-09-02T07:23:24.366 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:24 +0000 (0:00:00.042) 0:03:20.877 **** 2021-09-02T07:23:24.408 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-09-02T07:23:24.409 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:24.432 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:24 +0000 (0:00:00.064) 0:03:20.942 **** 2021-09-02T07:23:24.449 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:24.475 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:24 +0000 (0:00:00.041) 0:03:20.983 **** 2021-09-02T07:23:28.753 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-09-02T07:23:28.755 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64"]} 2021-09-02T07:23:28.777 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:28 +0000 (0:00:04.303) 0:03:25.287 **** 2021-09-02T07:23:28.963 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64"]} 2021-09-02T07:23:28.990 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:28 +0000 (0:00:00.210) 0:03:25.498 **** 2021-09-02T07:23:31.337 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-09-02T07:23:31.339 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-09-02T07:23:31.360 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:31 +0000 (0:00:02.372) 0:03:27.871 **** 2021-09-02T07:23:31.580 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-09-02T07:23:31.603 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:31 +0000 (0:00:00.242) 0:03:28.113 **** 2021-09-02T07:23:38.574 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-02T07:23:38.575 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.4.0+885+5e18b468.3.x86_64", "Installed: qemu-kvm-common-15:4.2.0-48.module_el8.4.0+885+5e18b468.3.x86_64", "Installed: qemu-kvm-core-15:4.2.0-48.module_el8.4.0+885+5e18b468.3.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:4.2.0-48.module_el8.4.0+885+5e18b468.3.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64", "Removed: qemu-img-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64"]} 2021-09-02T07:23:38.597 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:38 +0000 (0:00:06.993) 0:03:35.107 **** 2021-09-02T07:23:38.669 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:38.690 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:38 +0000 (0:00:00.092) 0:03:35.200 **** 2021-09-02T07:23:38.712 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-02T07:23:38.713 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.4.0+885+5e18b468.3.x86_64", "Installed: qemu-kvm-common-15:4.2.0-48.module_el8.4.0+885+5e18b468.3.x86_64", "Installed: qemu-kvm-core-15:4.2.0-48.module_el8.4.0+885+5e18b468.3.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:4.2.0-48.module_el8.4.0+885+5e18b468.3.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64", "Removed: qemu-img-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64"]} 2021-09-02T07:23:38.742 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:38 +0000 (0:00:00.051) 0:03:35.252 **** 2021-09-02T07:23:38.825 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-09-02T07:23:38.827 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:38.847 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:38 +0000 (0:00:00.105) 0:03:35.357 **** 2021-09-02T07:23:40.344 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-09-02T07:23:40.345 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-02T07:23:40.372 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:40 +0000 (0:00:01.524) 0:03:36.882 **** 2021-09-02T07:23:40.519 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-02T07:23:40.544 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:40 +0000 (0:00:00.172) 0:03:37.054 **** 2021-09-02T07:23:41.992 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-02T07:23:41.994 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-02T07:23:42.020 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:42 +0000 (0:00:01.475) 0:03:38.530 **** 2021-09-02T07:23:42.150 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-02T07:23:42.182 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:42 +0000 (0:00:00.161) 0:03:38.692 **** 2021-09-02T07:23:42.399 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-09-02T07:23:42.400 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveState": "inactive", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity 2021-09-02T07:23:42.401 INFO:teuthology.task.ansible.out:", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203592", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-09-02T07:23:42.426 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:42 +0000 (0:00:00.244) 0:03:38.936 **** 2021-09-02T07:23:42.576 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveState": "inactive", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity 2021-09-02T07:23:42.577 INFO:teuthology.task.ansible.out:", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127246", "LimitNPROCSoft": "127246", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127246", "LimitSIGPENDINGSoft": "127246", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203593", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-09-02T07:23:42.601 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:42 +0000 (0:00:00.175) 0:03:39.112 **** 2021-09-02T07:23:42.790 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ fatal: [smithi175.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-09-02T07:23:42.814 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:42 +0000 (0:00:00.211) 0:03:39.324 **** 2021-09-02T07:23:42.938 INFO:teuthology.task.ansible.out:fatal: [smithi197.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-09-02T07:23:42.960 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:42 +0000 (0:00:00.146) 0:03:39.470 **** 2021-09-02T07:23:43.461 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-09-02T07:23:43.463 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-09-02T07:23:43.488 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:43 +0000 (0:00:00.527) 0:03:39.998 **** 2021-09-02T07:23:43.508 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:43 +0000 (0:00:00.019) 0:03:40.018 **** 2021-09-02T07:23:43.557 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-09-02T07:23:43.559 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:43.580 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:43 +0000 (0:00:00.072) 0:03:40.090 **** 2021-09-02T07:23:43.587 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:43.612 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:43 +0000 (0:00:00.030) 0:03:40.121 **** 2021-09-02T07:23:43.644 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:43.668 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:43 +0000 (0:00:00.056) 0:03:40.178 **** 2021-09-02T07:23:43.685 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:43.708 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:43 +0000 (0:00:00.040) 0:03:40.218 **** 2021-09-02T07:23:43.735 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-09-02T07:23:43.737 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:43.758 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:43 +0000 (0:00:00.050) 0:03:40.269 **** 2021-09-02T07:23:43.776 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:43.799 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:43 +0000 (0:00:00.040) 0:03:40.309 **** 2021-09-02T07:23:43.827 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-09-02T07:23:43.828 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:43.851 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:43 +0000 (0:00:00.052) 0:03:40.361 **** 2021-09-02T07:23:43.870 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:43.896 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:43 +0000 (0:00:00.044) 0:03:40.406 **** 2021-09-02T07:23:43.935 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-09-02T07:23:43.937 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:43.959 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:43 +0000 (0:00:00.063) 0:03:40.470 **** 2021-09-02T07:23:43.978 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:44.004 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:43 +0000 (0:00:00.043) 0:03:40.514 **** 2021-09-02T07:23:44.035 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-09-02T07:23:44.037 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:44.060 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:44 +0000 (0:00:00.056) 0:03:40.570 **** 2021-09-02T07:23:44.080 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:44.114 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:44 +0000 (0:00:00.053) 0:03:40.624 **** 2021-09-02T07:23:44.135 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-09-02T07:23:44.162 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:44 +0000 (0:00:00.047) 0:03:40.672 **** 2021-09-02T07:23:44.214 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:44 +0000 (0:00:00.052) 0:03:40.725 **** 2021-09-02T07:23:44.232 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-09-02T07:23:44.234 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:44.259 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:44 +0000 (0:00:00.044) 0:03:40.770 **** 2021-09-02T07:23:44.279 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:44.305 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:44 +0000 (0:00:00.045) 0:03:40.815 **** 2021-09-02T07:23:44.336 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:44.359 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:44 +0000 (0:00:00.054) 0:03:40.870 **** 2021-09-02T07:23:44.382 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:44.413 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:44 +0000 (0:00:00.046) 0:03:40.916 **** 2021-09-02T07:23:44.446 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-09-02T07:23:44.449 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-09-02T07:23:44.480 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:44 +0000 (0:00:00.072) 0:03:40.988 **** 2021-09-02T07:23:44.495 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-09-02T07:23:44.519 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:44 +0000 (0:00:00.041) 0:03:41.029 **** 2021-09-02T07:23:44.558 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:44.582 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:44 +0000 (0:00:00.062) 0:03:41.092 **** 2021-09-02T07:23:44.601 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:44.630 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:44 +0000 (0:00:00.042) 0:03:41.134 **** 2021-09-02T07:23:44.653 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-09-02T07:23:44.685 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:44 +0000 (0:00:00.059) 0:03:41.194 **** 2021-09-02T07:23:44.717 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:44 +0000 (0:00:00.033) 0:03:41.227 **** 2021-09-02T07:23:44.747 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:44.771 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:44 +0000 (0:00:00.053) 0:03:41.281 **** 2021-09-02T07:23:44.798 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:44.825 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:44 +0000 (0:00:00.052) 0:03:41.334 **** 2021-09-02T07:23:44.854 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:44.877 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:44 +0000 (0:00:00.053) 0:03:41.388 **** 2021-09-02T07:23:44.896 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:44.920 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:44 +0000 (0:00:00.042) 0:03:41.430 **** 2021-09-02T07:23:44.939 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-09-02T07:23:44.941 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:44.964 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:44 +0000 (0:00:00.043) 0:03:41.474 **** 2021-09-02T07:23:44.996 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:45.020 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:45 +0000 (0:00:00.056) 0:03:41.530 **** 2021-09-02T07:23:45.037 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-02T07:23:45.038 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:45.062 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:45 +0000 (0:00:00.041) 0:03:41.572 **** 2021-09-02T07:23:45.090 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:45.114 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:45 +0000 (0:00:00.051) 0:03:41.624 **** 2021-09-02T07:23:45.132 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:45.156 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:45 +0000 (0:00:00.042) 0:03:41.666 **** 2021-09-02T07:23:45.184 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:45.207 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:45 +0000 (0:00:00.051) 0:03:41.717 **** 2021-09-02T07:23:45.235 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:45.262 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:45 +0000 (0:00:00.051) 0:03:41.769 **** 2021-09-02T07:23:45.279 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:45.306 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:45 +0000 (0:00:00.046) 0:03:41.815 **** 2021-09-02T07:23:45.345 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:45.371 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:45 +0000 (0:00:00.065) 0:03:41.881 **** 2021-09-02T07:23:45.373 INFO:teuthology.task.ansible.out: 2021-09-02T07:23:45.381 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:45.404 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:45 +0000 (0:00:00.033) 0:03:41.914 **** 2021-09-02T07:23:45.454 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-09-02T07:23:45.455 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:45.479 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:45 +0000 (0:00:00.074) 0:03:41.989 **** 2021-09-02T07:23:45.487 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:45.511 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:45 +0000 (0:00:00.032) 0:03:42.022 **** 2021-09-02T07:23:45.559 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:45.578 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:45 +0000 (0:00:00.066) 0:03:42.089 **** 2021-09-02T07:23:45.600 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:45.630 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:45 +0000 (0:00:00.051) 0:03:42.140 **** 2021-09-02T07:23:45.680 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-09-02T07:23:45.682 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:45.703 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:45 +0000 (0:00:00.073) 0:03:42.214 **** 2021-09-02T07:23:45.705 INFO:teuthology.task.ansible.out: 2021-09-02T07:23:45.713 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:45.739 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:45 +0000 (0:00:00.035) 0:03:42.249 **** 2021-09-02T07:23:45.783 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-09-02T07:23:45.787 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:45.808 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:45 +0000 (0:00:00.068) 0:03:42.318 **** 2021-09-02T07:23:45.817 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:45.846 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:45 +0000 (0:00:00.037) 0:03:42.356 **** 2021-09-02T07:23:45.884 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-09-02T07:23:45.886 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:45.912 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:45 +0000 (0:00:00.065) 0:03:42.422 **** 2021-09-02T07:23:45.931 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:45.954 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:45 +0000 (0:00:00.042) 0:03:42.464 **** 2021-09-02T07:23:45.992 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:46.017 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:46 +0000 (0:00:00.062) 0:03:42.527 **** 2021-09-02T07:23:46.037 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:46.061 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:46 +0000 (0:00:00.043) 0:03:42.571 **** 2021-09-02T07:23:46.090 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-09-02T07:23:46.092 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:46.125 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:46 +0000 (0:00:00.063) 0:03:42.634 **** 2021-09-02T07:23:46.134 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:46.160 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:46 +0000 (0:00:00.034) 0:03:42.668 **** 2021-09-02T07:23:46.187 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-09-02T07:23:46.191 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:46.211 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:46 +0000 (0:00:00.053) 0:03:42.722 **** 2021-09-02T07:23:46.240 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:46.264 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:46 +0000 (0:00:00.052) 0:03:42.774 **** 2021-09-02T07:23:46.288 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:46.311 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:46 +0000 (0:00:00.047) 0:03:42.822 **** 2021-09-02T07:23:46.341 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:46.365 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:46 +0000 (0:00:00.053) 0:03:42.875 **** 2021-09-02T07:23:46.383 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:46.408 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:46 +0000 (0:00:00.043) 0:03:42.918 **** 2021-09-02T07:23:46.437 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:46.460 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:46 +0000 (0:00:00.052) 0:03:42.971 **** 2021-09-02T07:23:46.480 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:46.504 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:46 +0000 (0:00:00.043) 0:03:43.014 **** 2021-09-02T07:23:46.533 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:46.557 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:46 +0000 (0:00:00.052) 0:03:43.066 **** 2021-09-02T07:23:46.574 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:46.599 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:46 +0000 (0:00:00.042) 0:03:43.109 **** 2021-09-02T07:23:46.632 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:46.657 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:46 +0000 (0:00:00.058) 0:03:43.167 **** 2021-09-02T07:23:46.675 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:46.699 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:46 +0000 (0:00:00.041) 0:03:43.209 **** 2021-09-02T07:23:46.728 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:46.763 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:46 +0000 (0:00:00.063) 0:03:43.272 **** 2021-09-02T07:23:46.789 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:46.822 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:46 +0000 (0:00:00.059) 0:03:43.332 **** 2021-09-02T07:23:46.878 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:46.902 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:46 +0000 (0:00:00.079) 0:03:43.412 **** 2021-09-02T07:23:46.908 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-09-02T07:23:46.911 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:46.933 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:46 +0000 (0:00:00.031) 0:03:43.443 **** 2021-09-02T07:23:46.987 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:47.002 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:46 +0000 (0:00:00.067) 0:03:43.511 **** 2021-09-02T07:23:47.024 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-09-02T07:23:47.031 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:47.054 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:47 +0000 (0:00:00.053) 0:03:43.565 **** 2021-09-02T07:23:47.088 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:47.117 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:47 +0000 (0:00:00.062) 0:03:43.627 **** 2021-09-02T07:23:47.147 INFO:teuthology.task.ansible.out: 2021-09-02T07:23:47.148 INFO:teuthology.task.ansible.out:TASK [testnode : set_fact] ***************************************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:47.174 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:47 +0000 (0:00:00.055) 0:03:43.683 **** 2021-09-02T07:23:47.193 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:47.223 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:47 +0000 (0:00:00.046) 0:03:43.729 **** 2021-09-02T07:23:47.282 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-09-02T07:23:47.284 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:47.309 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:47 +0000 (0:00:00.089) 0:03:43.818 **** 2021-09-02T07:23:47.326 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:47.352 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:47 +0000 (0:00:00.043) 0:03:43.862 **** 2021-09-02T07:23:47.381 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-02T07:23:47.383 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:47.408 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:47 +0000 (0:00:00.054) 0:03:43.917 **** 2021-09-02T07:23:47.428 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:47.452 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:47 +0000 (0:00:00.044) 0:03:43.962 **** 2021-09-02T07:23:47.480 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-02T07:23:47.482 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:47.504 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:47 +0000 (0:00:00.052) 0:03:44.014 **** 2021-09-02T07:23:47.530 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:47.557 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:47 +0000 (0:00:00.052) 0:03:44.067 **** 2021-09-02T07:23:47.586 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:47.609 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:47 +0000 (0:00:00.052) 0:03:44.120 **** 2021-09-02T07:23:47.664 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:47.694 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:47 +0000 (0:00:00.079) 0:03:44.199 **** 2021-09-02T07:23:47.900 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-09-02T07:23:47.928 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:47 +0000 (0:00:00.238) 0:03:44.438 **** 2021-09-02T07:23:48.004 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-09-02T07:23:48.038 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:48 +0000 (0:00:00.110) 0:03:44.548 **** 2021-09-02T07:23:48.202 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-02T07:23:48.225 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:48 +0000 (0:00:00.186) 0:03:44.735 **** 2021-09-02T07:23:48.321 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-02T07:23:48.345 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:48 +0000 (0:00:00.120) 0:03:44.856 **** 2021-09-02T07:23:48.494 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** [WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. 2021-09-02T07:23:48.496 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-09-02T07:23:48.518 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:48 +0000 (0:00:00.172) 0:03:45.028 **** 2021-09-02T07:23:48.594 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:48.622 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:48 +0000 (0:00:00.103) 0:03:45.132 **** 2021-09-02T07:23:48.643 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:48 +0000 (0:00:00.021) 0:03:45.153 **** 2021-09-02T07:23:48.682 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-09-02T07:23:48.684 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:48.710 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:48 +0000 (0:00:00.067) 0:03:45.221 **** 2021-09-02T07:23:48.719 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-09-02T07:23:48.721 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:48.744 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:48 +0000 (0:00:00.033) 0:03:45.254 **** 2021-09-02T07:23:48.793 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:48.830 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:48 +0000 (0:00:00.086) 0:03:45.340 **** 2021-09-02T07:23:48.839 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-09-02T07:23:48.841 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:48.864 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:48 +0000 (0:00:00.034) 0:03:45.374 **** 2021-09-02T07:23:48.894 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:48.917 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:48 +0000 (0:00:00.052) 0:03:45.427 **** 2021-09-02T07:23:48.936 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-09-02T07:23:48.939 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:48.962 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:48 +0000 (0:00:00.044) 0:03:45.472 **** 2021-09-02T07:23:48.982 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.009 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.047) 0:03:45.519 **** 2021-09-02T07:23:49.011 INFO:teuthology.task.ansible.out: 2021-09-02T07:23:49.031 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.057 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.048) 0:03:45.567 **** 2021-09-02T07:23:49.060 INFO:teuthology.task.ansible.out: 2021-09-02T07:23:49.076 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.106 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.047) 0:03:45.615 **** 2021-09-02T07:23:49.125 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.149 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.043) 0:03:45.658 **** 2021-09-02T07:23:49.175 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-02T07:23:49.177 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.202 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.053) 0:03:45.712 **** 2021-09-02T07:23:49.222 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.245 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.043) 0:03:45.755 **** 2021-09-02T07:23:49.273 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-02T07:23:49.275 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.297 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.051) 0:03:45.807 **** 2021-09-02T07:23:49.322 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.341 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.043) 0:03:45.851 **** 2021-09-02T07:23:49.344 INFO:teuthology.task.ansible.out: 2021-09-02T07:23:49.370 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.393 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.052) 0:03:45.904 **** 2021-09-02T07:23:49.412 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-02T07:23:49.414 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.436 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.043) 0:03:45.947 **** 2021-09-02T07:23:49.466 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.491 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.054) 0:03:46.001 **** 2021-09-02T07:23:49.521 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-02T07:23:49.524 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.546 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.054) 0:03:46.056 **** 2021-09-02T07:23:49.563 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-09-02T07:23:49.565 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.590 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.044) 0:03:46.100 **** 2021-09-02T07:23:49.620 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-09-02T07:23:49.622 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.645 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.055) 0:03:46.155 **** 2021-09-02T07:23:49.663 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-09-02T07:23:49.665 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.687 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.042) 0:03:46.198 **** 2021-09-02T07:23:49.717 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.741 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.052) 0:03:46.250 **** 2021-09-02T07:23:49.769 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.796 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.055) 0:03:46.305 **** 2021-09-02T07:23:49.814 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.841 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.044) 0:03:46.350 **** 2021-09-02T07:23:49.867 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-09-02T07:23:49.871 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.893 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.053) 0:03:46.404 **** 2021-09-02T07:23:49.913 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.940 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.046) 0:03:46.450 **** 2021-09-02T07:23:49.967 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-09-02T07:23:49.969 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:49.992 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:49 +0000 (0:00:00.051) 0:03:46.502 **** 2021-09-02T07:23:50.009 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-09-02T07:23:50.011 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:50.033 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:50 +0000 (0:00:00.041) 0:03:46.543 **** 2021-09-02T07:23:50.060 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:50.084 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:50 +0000 (0:00:00.050) 0:03:46.594 **** 2021-09-02T07:23:50.103 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-09-02T07:23:50.104 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:50.128 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:50 +0000 (0:00:00.044) 0:03:46.638 **** 2021-09-02T07:23:50.157 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:50.180 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:50 +0000 (0:00:00.051) 0:03:46.690 **** 2021-09-02T07:23:50.197 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-09-02T07:23:50.198 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:50.222 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:50 +0000 (0:00:00.042) 0:03:46.732 **** 2021-09-02T07:23:50.240 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-09-02T07:23:50.242 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:50.268 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:50 +0000 (0:00:00.045) 0:03:46.778 **** 2021-09-02T07:23:50.289 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:50.316 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:50 +0000 (0:00:00.045) 0:03:46.823 **** 2021-09-02T07:23:50.331 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-09-02T07:23:50.333 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:50.356 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:50 +0000 (0:00:00.042) 0:03:46.866 **** 2021-09-02T07:23:50.384 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:50.408 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:50 +0000 (0:00:00.051) 0:03:46.918 **** 2021-09-02T07:23:50.427 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:50.449 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:50 +0000 (0:00:00.042) 0:03:46.960 **** 2021-09-02T07:23:50.483 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:50.503 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:50 +0000 (0:00:00.053) 0:03:47.013 **** 2021-09-02T07:23:50.525 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:50.557 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:50 +0000 (0:00:00.053) 0:03:47.067 **** 2021-09-02T07:23:50.586 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:50.612 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:50 +0000 (0:00:00.054) 0:03:47.122 **** 2021-09-02T07:23:50.638 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-09-02T07:23:50.641 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:50.664 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:50 +0000 (0:00:00.051) 0:03:47.173 **** 2021-09-02T07:23:50.682 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-09-02T07:23:50.684 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:50.707 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:50 +0000 (0:00:00.043) 0:03:47.217 **** 2021-09-02T07:23:50.757 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-02T07:23:50.758 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-09-02T07:23:50.782 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:50 +0000 (0:00:00.075) 0:03:47.292 **** 2021-09-02T07:23:50.790 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-09-02T07:23:50.792 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:50.818 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:50 +0000 (0:00:00.034) 0:03:47.327 **** 2021-09-02T07:23:50.875 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi175.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-09-02T07:23:50.886 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-09-02T07:23:50.897 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-09-02T07:23:50.899 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-09-02T07:23:50.912 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-09-02T07:23:50.914 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi197.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-09-02T07:23:50.938 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:50 +0000 (0:00:00.120) 0:03:47.448 **** 2021-09-02T07:23:50.951 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi175.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-09-02T07:23:51.000 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:50 +0000 (0:00:00.062) 0:03:47.510 **** 2021-09-02T07:23:51.056 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-09-02T07:23:51.068 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-09-02T07:23:51.070 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-09-02T07:23:51.091 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-09-02T07:23:51.094 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-09-02T07:23:51.095 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-09-02T07:23:51.139 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:51 +0000 (0:00:00.138) 0:03:47.649 **** 2021-09-02T07:23:51.321 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.008158", "end": "2021-09-02 07:23:51.300145", "rc": 0, "start": "2021-09-02 07:23:51.291987", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/virtualenv/bin/teuthology-lock"]} 2021-09-02T07:23:51.356 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:51 +0000 (0:00:00.217) 0:03:47.866 **** 2021-09-02T07:23:51.358 INFO:teuthology.task.ansible.out: 2021-09-02T07:23:51.438 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.006385", "end": "2021-09-02 07:23:51.403526", "rc": 0, "start": "2021-09-02 07:23:51.397141", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/virtualenv/bin/teuthology-lock"]} 2021-09-02T07:23:51.465 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:51 +0000 (0:00:00.107) 0:03:47.974 **** 2021-09-02T07:23:51.470 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-09-02T07:23:51.472 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:51.497 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:51 +0000 (0:00:00.033) 0:03:48.008 **** 2021-09-02T07:23:51.527 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:51.554 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:51 +0000 (0:00:00.055) 0:03:48.064 **** 2021-09-02T07:23:51.583 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:51.610 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:51 +0000 (0:00:00.056) 0:03:48.120 **** 2021-09-02T07:23:51.629 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:51.655 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:51 +0000 (0:00:00.045) 0:03:48.166 **** 2021-09-02T07:23:51.685 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:51.710 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:51 +0000 (0:00:00.053) 0:03:48.219 **** 2021-09-02T07:23:51.738 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:51.764 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:51 +0000 (0:00:00.054) 0:03:48.274 **** 2021-09-02T07:23:51.782 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-09-02T07:23:51.784 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:51.813 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:51 +0000 (0:00:00.048) 0:03:48.322 **** 2021-09-02T07:23:51.840 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:51.878 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:51 +0000 (0:00:00.063) 0:03:48.386 **** 2021-09-02T07:23:53.443 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-02T07:23:53.463 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:53 +0000 (0:00:01.587) 0:03:49.973 **** 2021-09-02T07:23:53.535 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-02T07:23:53.537 INFO:teuthology.task.ansible.out: 2021-09-02T07:23:53.561 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:53 +0000 (0:00:00.097) 0:03:50.071 **** 2021-09-02T07:23:53.571 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [smithi175.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930259316736, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227114091, 'block_used': 12981017, 'inode_total': 61054976, 'inode_available': 60985860, 'inode_used': 69116, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227114091, "block_size": 4096, "block_total": 240095108, "block_used": 12981017, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985860, "inode_total": 61054976, "inode_used": 69116, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930259316736, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}} 2021-09-02T07:23:53.595 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:53 +0000 (0:00:00.033) 0:03:50.105 **** 2021-09-02T07:23:53.646 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930259316736, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227114091, 'block_used': 12981017, 'inode_total': 61054976, 'inode_available': 60985860, 'inode_used': 69116, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227114091, "block_size": 4096, "block_total": 240095108, "block_used": 12981017, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985860, "inode_total": 61054976, "inode_used": 69116, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930259316736, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}} 2021-09-02T07:23:53.670 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:53 +0000 (0:00:00.074) 0:03:50.180 **** 2021-09-02T07:23:53.824 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-09-02T07:23:53.826 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005008", "end": "2021-09-02 07:23:53.807480", "rc": 0, "start": "2021-09-02 07:23:53.802472", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:23:53.848 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:53 +0000 (0:00:00.178) 0:03:50.359 **** 2021-09-02T07:23:53.900 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005140", "end": "2021-09-02 07:23:53.879031", "rc": 0, "start": "2021-09-02 07:23:53.873891", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:23:53.924 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:53 +0000 (0:00:00.075) 0:03:50.434 **** 2021-09-02T07:23:53.931 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-09-02T07:23:53.933 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930259316736, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227114091, 'block_used': 12981017, 'inode_total': 61054976, 'inode_available': 60985860, 'inode_used': 69116, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227114091, "block_size": 4096, "block_total": 240095108, "block_used": 12981017, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985860, "inode_total": 61054976, "inode_used": 69116, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930259316736, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:53.960 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:53 +0000 (0:00:00.033) 0:03:50.468 **** 2021-09-02T07:23:53.998 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930259316736, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227114091, 'block_used': 12981017, 'inode_total': 61054976, 'inode_available': 60985860, 'inode_used': 69116, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227114091, "block_size": 4096, "block_total": 240095108, "block_used": 12981017, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985860, "inode_total": 61054976, "inode_used": 69116, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930259316736, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:23:54.025 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:54 +0000 (0:00:00.066) 0:03:50.534 **** 2021-09-02T07:23:54.031 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-09-02T07:23:54.059 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:54 +0000 (0:00:00.032) 0:03:50.566 **** 2021-09-02T07:23:54.117 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:23:54 +0000 (0:00:00.061) 0:03:50.628 **** 2021-09-02T07:23:55.594 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-09-02T07:23:55.595 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-eexRlw1qMMR9a7m3WxFnHdYqfrJou2NuVlamBh6dBSJ0RpGaXbirnUd1EKn2Oyew'], 'uuids': ['77598692-78dc-411b-a120-298242e945dc'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.268914", "end": "2021-09-02 07:23:55.563614", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-eexRlw1qMMR9a7m3WxFnHdYqfrJou2NuVlamBh6dBSJ0RpGaXbirnUd1EKn2Oyew"], "labels": [], "masters": [], "uuids": ["77598692-78dc-411b-a120-298242e945dc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:23:54.294700", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-02T07:23:55.732 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-i0GWiLUvWtQZ0xLj7vRmQJkt6QezKNt6Mcu0fV302M9Q1dq8Rz7LiYA6BjUcvW66'], 'uuids': ['2161bda9-d7c5-4e6f-a224-5fd64a4f01c7'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.355675", "end": "2021-09-02 07:23:55.704977", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-i0GWiLUvWtQZ0xLj7vRmQJkt6QezKNt6Mcu0fV302M9Q1dq8Rz7LiYA6BjUcvW66"], "labels": [], "masters": [], "uuids": ["2161bda9-d7c5-4e6f-a224-5fd64a4f01c7"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:23:54.349302", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-02T07:23:56.846 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-4xkFs2-onZ1-2S1B-DDK1-Rrn7-im81-KqXw6d', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300M0400BGN', 'nvme-nvme.8086-435646543632333330304d3034303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.062685", "end": "2021-09-02 07:23:56.817848", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-4xkFs2-onZ1-2S1B-DDK1-Rrn7-im81-KqXw6d", "nvme-INTEL_SSDPEDMD400G4_CVFT623300M0400BGN", "nvme-nvme.8086-435646543632333330304d3034303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:23:55.755163", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-09-02T07:23:56.848 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-02T07:23:56.995 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-jlhLyR-0e8H-Ukb7-o7jr-CauF-sdFA-Re1wif', 'nvme-INTEL_SSDPEDMD400G4_PHFT620500HG400BGN', 'nvme-nvme.8086-50484654363230353030484734303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.080602", "end": "2021-09-02 07:23:56.970451", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-jlhLyR-0e8H-Ukb7-o7jr-CauF-sdFA-Re1wif", "nvme-INTEL_SSDPEDMD400G4_PHFT620500HG400BGN", "nvme-nvme.8086-50484654363230353030484734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:23:55.889849", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-09-02T07:23:56.997 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-02T07:23:58.085 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-eexRlw1qMMR9a7m3WxFnHdYqfrJou2Nu5b3pN3fvzIm9NCK30k1M3lE6SHFZHl5P'], 'uuids': ['361540f2-9a6a-4ae4-891f-6173dfe9c25a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.053175", "end": "2021-09-02 07:23:58.054403", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-eexRlw1qMMR9a7m3WxFnHdYqfrJou2Nu5b3pN3fvzIm9NCK30k1M3lE6SHFZHl5P"], "labels": [], "masters": [], "uuids": ["361540f2-9a6a-4ae4-891f-6173dfe9c25a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:23:57.001228", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-02T07:23:58.263 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-i0GWiLUvWtQZ0xLj7vRmQJkt6QezKNt6FwydhM8wu8NdYuGXlbjjkeTavVxwK7wW'], 'uuids': ['296dc2dc-ed61-46e2-aef5-3565428f876b'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.072202", "end": "2021-09-02 07:23:58.231758", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-i0GWiLUvWtQZ0xLj7vRmQJkt6QezKNt6FwydhM8wu8NdYuGXlbjjkeTavVxwK7wW"], "labels": [], "masters": [], "uuids": ["296dc2dc-ed61-46e2-aef5-3565428f876b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:23:57.159556", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-02T07:23:59.313 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-eexRlw1qMMR9a7m3WxFnHdYqfrJou2Nud6bK7RH3B5uBND9OA5i07hYv4XJ3IE4I'], 'uuids': ['9ff9193a-fe90-429b-be37-c784e684cb31'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.040934", "end": "2021-09-02 07:23:59.281068", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-eexRlw1qMMR9a7m3WxFnHdYqfrJou2Nud6bK7RH3B5uBND9OA5i07hYv4XJ3IE4I"], "labels": [], "masters": [], "uuids": ["9ff9193a-fe90-429b-be37-c784e684cb31"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:23:58.240134", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-02T07:23:59.512 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-i0GWiLUvWtQZ0xLj7vRmQJkt6QezKNt6fJ1vV3qaWWd48kXlJYZHUsm8yYKx8m8u'], 'uuids': ['127621a7-578c-4869-82ee-bcd5f3c68dc5'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.060149", "end": "2021-09-02 07:23:59.479143", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-i0GWiLUvWtQZ0xLj7vRmQJkt6QezKNt6fJ1vV3qaWWd48kXlJYZHUsm8yYKx8m8u"], "labels": [], "masters": [], "uuids": ["127621a7-578c-4869-82ee-bcd5f3c68dc5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:23:58.418994", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-02T07:24:00.550 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-eexRlw1qMMR9a7m3WxFnHdYqfrJou2NuPBtXORthQA1K0hRszCGmeRf5mzXx62l5'], 'uuids': ['385f18a4-a8b0-473d-9e04-feb64d8af722'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.056722", "end": "2021-09-02 07:24:00.522364", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-eexRlw1qMMR9a7m3WxFnHdYqfrJou2NuPBtXORthQA1K0hRszCGmeRf5mzXx62l5"], "labels": [], "masters": [], "uuids": ["385f18a4-a8b0-473d-9e04-feb64d8af722"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:23:59.465642", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-02T07:24:00.552 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XJFL', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XJFL', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XJFL', 'scsi-35000c5009292c947', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XJFL', 'wwn-0x5000c5009292c947'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XJFL', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c5009292c947', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XJFL-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XJFL-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XJFL-part1', 'scsi-35000c5009292c947-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XJFL-part1', 'wwn-0x5000c5009292c947-part1'], 'uuids': ['fcc12580-8f3b-4cec-aef7-e7427fa239b7'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XJFL", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XJFL", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XJFL", "scsi-35000c5009292c947", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XJFL", "wwn-0x5000c5009292c947"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XJFL-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XJFL-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-09-02T07:24:00.553 INFO:teuthology.task.ansible.out:Z1W5XJFL-part1", "scsi-35000c5009292c947-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XJFL-part1", "wwn-0x5000c5009292c947-part1"], "labels": [], "masters": [], "uuids": ["fcc12580-8f3b-4cec-aef7-e7427fa239b7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XJFL", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009292c947"}}, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:00.773 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-i0GWiLUvWtQZ0xLj7vRmQJkt6QezKNt6BlJSPkkztNylXMCtPNFwkl4CQgjgQ0Wx'], 'uuids': ['db7adee5-9f87-4992-bdc4-0e442703a9db'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.075743", "end": "2021-09-02 07:24:00.740590", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-i0GWiLUvWtQZ0xLj7vRmQJkt6QezKNt6BlJSPkkztNylXMCtPNFwkl4CQgjgQ0Wx"], "labels": [], "masters": [], "uuids": ["db7adee5-9f87-4992-bdc4-0e442703a9db"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:23:59.664847", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-02T07:24:00.785 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XK3R', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XK3R', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XK3R', 'scsi-35000c5009293d217', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XK3R', 'wwn-0x5000c5009293d217'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XK3R', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c5009293d217', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XK3R-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XK3R-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XK3R-part1', 'scsi-35000c5009293d217-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XK3R-part1', 'wwn-0x5000c5009293d217-part1'], 'uuids': ['fcc12580-8f3b-4cec-aef7-e7427fa239b7'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XK3R", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XK3R", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XK3R", "scsi-35000c5009293d217", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XK3R", "wwn-0x5000c5009293d217"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XK3R-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XK3R-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-09-02T07:24:00.787 INFO:teuthology.task.ansible.out:Z1W5XK3R-part1", "scsi-35000c5009293d217-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XK3R-part1", "wwn-0x5000c5009293d217-part1"], "labels": [], "masters": [], "uuids": ["fcc12580-8f3b-4cec-aef7-e7427fa239b7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XK3R", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009293d217"}}, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:01.790 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-eexRlw1qMMR9a7m3WxFnHdYqfrJou2Nu1yvcU3c31ayuMS2RtRVHhJJvRqS7ajAb'], 'uuids': ['8bcb3982-f445-46db-bbca-eb4d2a784dba'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.052119", "end": "2021-09-02 07:24:01.763941", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-eexRlw1qMMR9a7m3WxFnHdYqfrJou2Nu1yvcU3c31ayuMS2RtRVHhJJvRqS7ajAb"], "labels": [], "masters": [], "uuids": ["8bcb3982-f445-46db-bbca-eb4d2a784dba"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:24:00.711822", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-02T07:24:01.819 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:01 +0000 (0:00:07.702) 0:03:58.330 **** 2021-09-02T07:24:02.016 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-i0GWiLUvWtQZ0xLj7vRmQJkt6QezKNt6FwAzb7JO1URJ6fniLfmvLl8FJNPzTiTT'], 'uuids': ['5374cf37-82d0-4b01-a116-61c4d47be6a1'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.049811", "end": "2021-09-02 07:24:01.982043", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-i0GWiLUvWtQZ0xLj7vRmQJkt6QezKNt6FwAzb7JO1URJ6fniLfmvLl8FJNPzTiTT"], "labels": [], "masters": [], "uuids": ["5374cf37-82d0-4b01-a116-61c4d47be6a1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:24:00.932232", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-02T07:24:02.045 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:02 +0000 (0:00:00.225) 0:03:58.556 **** 2021-09-02T07:24:02.369 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-09-02T07:24:02.371 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-eexRlw1qMMR9a7m3WxFnHdYqfrJou2NuVlamBh6dBSJ0RpGaXbirnUd1EKn2Oyew'], 'uuids': ['77598692-78dc-411b-a120-298242e945dc'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.142881", "end": "2021-09-02 07:24:02.347906", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-eexRlw1qMMR9a7m3WxFnHdYqfrJou2NuVlamBh6dBSJ0RpGaXbirnUd1EKn2Oyew"], "labels": [], "masters": [], "uuids": ["77598692-78dc-411b-a120-298242e945dc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:24:02.205025", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140197 s, 823 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140197 s, 823 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:02.455 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-i0GWiLUvWtQZ0xLj7vRmQJkt6QezKNt6Mcu0fV302M9Q1dq8Rz7LiYA6BjUcvW66'], 'uuids': ['2161bda9-d7c5-4e6f-a224-5fd64a4f01c7'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.143257", "end": "2021-09-02 07:24:02.423108", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-i0GWiLUvWtQZ0xLj7vRmQJkt6QezKNt6Mcu0fV302M9Q1dq8Rz7LiYA6BjUcvW66"], "labels": [], "masters": [], "uuids": ["2161bda9-d7c5-4e6f-a224-5fd64a4f01c7"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:24:02.279851", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140498 s, 821 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140498 s, 821 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:02.592 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-4xkFs2-onZ1-2S1B-DDK1-Rrn7-im81-KqXw6d', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300M0400BGN', 'nvme-nvme.8086-435646543632333330304d3034303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039341", "end": "2021-09-02 07:24:02.560801", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-4xkFs2-onZ1-2S1B-DDK1-Rrn7-im81-KqXw6d", "nvme-INTEL_SSDPEDMD400G4_CVFT623300M0400BGN", "nvme-nvme.8086-435646543632333330304d3034303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:24:02.521460", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0366867 s, 3.1 GB/s", "std 2021-09-02T07:24:02.594 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0366867 s, 3.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:02.667 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-jlhLyR-0e8H-Ukb7-o7jr-CauF-sdFA-Re1wif', 'nvme-INTEL_SSDPEDMD400G4_PHFT620500HG400BGN', 'nvme-nvme.8086-50484654363230353030484734303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039957", "end": "2021-09-02 07:24:02.635733", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-jlhLyR-0e8H-Ukb7-o7jr-CauF-sdFA-Re1wif", "nvme-INTEL_SSDPEDMD400G4_PHFT620500HG400BGN", "nvme-nvme.8086-50484654363230353030484734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:24:02.595776", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0373607 s, 3.1 GB/s", "std 2021-09-02T07:24:02.669 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0373607 s, 3.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:02.910 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-eexRlw1qMMR9a7m3WxFnHdYqfrJou2Nu5b3pN3fvzIm9NCK30k1M3lE6SHFZHl5P'], 'uuids': ['361540f2-9a6a-4ae4-891f-6173dfe9c25a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.146003", "end": "2021-09-02 07:24:02.886134", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-eexRlw1qMMR9a7m3WxFnHdYqfrJou2Nu5b3pN3fvzIm9NCK30k1M3lE6SHFZHl5P"], "labels": [], "masters": [], "uuids": ["361540f2-9a6a-4ae4-891f-6173dfe9c25a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:24:02.740131", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.143355 s, 805 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.143355 s, 805 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:03.006 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-i0GWiLUvWtQZ0xLj7vRmQJkt6QezKNt6FwydhM8wu8NdYuGXlbjjkeTavVxwK7wW'], 'uuids': ['296dc2dc-ed61-46e2-aef5-3565428f876b'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.151620", "end": "2021-09-02 07:24:02.970120", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-i0GWiLUvWtQZ0xLj7vRmQJkt6QezKNt6FwydhM8wu8NdYuGXlbjjkeTavVxwK7wW"], "labels": [], "masters": [], "uuids": ["296dc2dc-ed61-46e2-aef5-3565428f876b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:24:02.818500", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148967 s, 774 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148967 s, 774 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:03.225 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-eexRlw1qMMR9a7m3WxFnHdYqfrJou2Nud6bK7RH3B5uBND9OA5i07hYv4XJ3IE4I'], 'uuids': ['9ff9193a-fe90-429b-be37-c784e684cb31'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.143723", "end": "2021-09-02 07:24:03.204434", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-eexRlw1qMMR9a7m3WxFnHdYqfrJou2Nud6bK7RH3B5uBND9OA5i07hYv4XJ3IE4I"], "labels": [], "masters": [], "uuids": ["9ff9193a-fe90-429b-be37-c784e684cb31"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:24:03.060711", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.141084 s, 818 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.141084 s, 818 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:03.299 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-i0GWiLUvWtQZ0xLj7vRmQJkt6QezKNt6fJ1vV3qaWWd48kXlJYZHUsm8yYKx8m8u'], 'uuids': ['127621a7-578c-4869-82ee-bcd5f3c68dc5'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.141429", "end": "2021-09-02 07:24:03.280271", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-i0GWiLUvWtQZ0xLj7vRmQJkt6QezKNt6fJ1vV3qaWWd48kXlJYZHUsm8yYKx8m8u"], "labels": [], "masters": [], "uuids": ["127621a7-578c-4869-82ee-bcd5f3c68dc5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:24:03.138842", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.13899 s, 830 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.13899 s, 830 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:03.545 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-eexRlw1qMMR9a7m3WxFnHdYqfrJou2NuPBtXORthQA1K0hRszCGmeRf5mzXx62l5'], 'uuids': ['385f18a4-a8b0-473d-9e04-feb64d8af722'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.145258", "end": "2021-09-02 07:24:03.521598", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-eexRlw1qMMR9a7m3WxFnHdYqfrJou2NuPBtXORthQA1K0hRszCGmeRf5mzXx62l5"], "labels": [], "masters": [], "uuids": ["385f18a4-a8b0-473d-9e04-feb64d8af722"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:24:03.376340", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142617 s, 809 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142617 s, 809 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:03.557 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XJFL', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XJFL', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XJFL', 'scsi-35000c5009292c947', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XJFL', 'wwn-0x5000c5009292c947'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XJFL', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c5009292c947', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XJFL-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XJFL-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XJFL-part1', 'scsi-35000c5009292c947-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XJFL-part1', 'wwn-0x5000c5009292c947-part1'], 'uuids': ['fcc12580-8f3b-4cec-aef7-e7427fa239b7'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XJFL", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XJFL", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XJFL", "scsi-35000c5009292c947", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XJFL", "wwn-0x5000c5009292c947"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XJFL-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XJFL-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-09-02T07:24:03.559 INFO:teuthology.task.ansible.out:Z1W5XJFL-part1", "scsi-35000c5009292c947-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XJFL-part1", "wwn-0x5000c5009292c947-part1"], "labels": [], "masters": [], "uuids": ["fcc12580-8f3b-4cec-aef7-e7427fa239b7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XJFL", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009292c947"}}, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:03.625 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-i0GWiLUvWtQZ0xLj7vRmQJkt6QezKNt6BlJSPkkztNylXMCtPNFwkl4CQgjgQ0Wx'], 'uuids': ['db7adee5-9f87-4992-bdc4-0e442703a9db'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.143224", "end": "2021-09-02 07:24:03.594479", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-i0GWiLUvWtQZ0xLj7vRmQJkt6QezKNt6BlJSPkkztNylXMCtPNFwkl4CQgjgQ0Wx"], "labels": [], "masters": [], "uuids": ["db7adee5-9f87-4992-bdc4-0e442703a9db"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:24:03.451255", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140791 s, 819 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140791 s, 819 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:03.640 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XK3R', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XK3R', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XK3R', 'scsi-35000c5009293d217', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XK3R', 'wwn-0x5000c5009293d217'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XK3R', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c5009293d217', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XK3R-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XK3R-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XK3R-part1', 'scsi-35000c5009293d217-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XK3R-part1', 'wwn-0x5000c5009293d217-part1'], 'uuids': ['fcc12580-8f3b-4cec-aef7-e7427fa239b7'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XK3R", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XK3R", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XK3R", "scsi-35000c5009293d217", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XK3R", "wwn-0x5000c5009293d217"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XK3R-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XK3R-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-09-02T07:24:03.642 INFO:teuthology.task.ansible.out:Z1W5XK3R-part1", "scsi-35000c5009293d217-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XK3R-part1", "wwn-0x5000c5009293d217-part1"], "labels": [], "masters": [], "uuids": ["fcc12580-8f3b-4cec-aef7-e7427fa239b7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XK3R", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009293d217"}}, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:03.873 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-eexRlw1qMMR9a7m3WxFnHdYqfrJou2Nu1yvcU3c31ayuMS2RtRVHhJJvRqS7ajAb'], 'uuids': ['8bcb3982-f445-46db-bbca-eb4d2a784dba'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.143969", "end": "2021-09-02 07:24:03.848319", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-eexRlw1qMMR9a7m3WxFnHdYqfrJou2Nu1yvcU3c31ayuMS2RtRVHhJJvRqS7ajAb"], "labels": [], "masters": [], "uuids": ["8bcb3982-f445-46db-bbca-eb4d2a784dba"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:24:03.704350", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.1414 s, 816 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.1414 s, 816 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:03.904 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:03 +0000 (0:00:01.855) 0:04:00.411 **** 2021-09-02T07:24:03.951 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-i0GWiLUvWtQZ0xLj7vRmQJkt6QezKNt6FwAzb7JO1URJ6fniLfmvLl8FJNPzTiTT'], 'uuids': ['5374cf37-82d0-4b01-a116-61c4d47be6a1'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.140647", "end": "2021-09-02 07:24:03.926754", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-i0GWiLUvWtQZ0xLj7vRmQJkt6QezKNt6FwAzb7JO1URJ6fniLfmvLl8FJNPzTiTT"], "labels": [], "masters": [], "uuids": ["5374cf37-82d0-4b01-a116-61c4d47be6a1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-02 07:24:03.786107", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138161 s, 835 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138161 s, 835 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:03.982 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:03 +0000 (0:00:00.081) 0:04:00.493 **** 2021-09-02T07:24:04.180 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi175.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.052276", "end": "2021-09-02 07:24:04.114634", "rc": 0, "start": "2021-09-02 07:24:04.062358", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:04.203 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:04 +0000 (0:00:00.220) 0:04:00.713 **** 2021-09-02T07:24:04.264 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.055229", "end": "2021-09-02 07:24:04.202081", "rc": 0, "start": "2021-09-02 07:24:04.146852", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:04.289 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:04 +0000 (0:00:00.083) 0:04:00.797 **** 2021-09-02T07:24:04.441 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-09-02T07:24:04.442 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.045869", "end": "2021-09-02 07:24:04.418544", "rc": 0, "start": "2021-09-02 07:24:04.372675", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:04.465 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:04 +0000 (0:00:00.178) 0:04:00.975 **** 2021-09-02T07:24:04.514 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.042412", "end": "2021-09-02 07:24:04.494172", "rc": 0, "start": "2021-09-02 07:24:04.451760", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:04.537 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:04 +0000 (0:00:00.072) 0:04:01.047 **** 2021-09-02T07:24:04.544 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-09-02T07:24:04.618 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:04 +0000 (0:00:00.077) 0:04:01.124 **** 2021-09-02T07:24:04.788 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:04 +0000 (0:00:00.173) 0:04:01.298 **** 2021-09-02T07:24:04.795 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-09-02T07:24:04.797 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:04.824 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:04 +0000 (0:00:00.036) 0:04:01.334 **** 2021-09-02T07:24:04.831 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:04.858 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:04 +0000 (0:00:00.032) 0:04:01.367 **** 2021-09-02T07:24:04.866 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:04.888 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:04 +0000 (0:00:00.031) 0:04:01.399 **** 2021-09-02T07:24:04.895 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:04.919 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:04 +0000 (0:00:00.030) 0:04:01.429 **** 2021-09-02T07:24:04.926 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:04.951 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:04 +0000 (0:00:00.032) 0:04:01.462 **** 2021-09-02T07:24:04.958 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:04.986 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:04 +0000 (0:00:00.034) 0:04:01.496 **** 2021-09-02T07:24:04.995 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:05.019 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:05 +0000 (0:00:00.033) 0:04:01.530 **** 2021-09-02T07:24:05.027 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:05.052 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:05 +0000 (0:00:00.032) 0:04:01.562 **** 2021-09-02T07:24:05.207 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi175.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-02T07:24:05.241 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:05 +0000 (0:00:00.187) 0:04:01.750 **** 2021-09-02T07:24:05.260 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-02T07:24:05.285 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:05 +0000 (0:00:00.043) 0:04:01.794 **** 2021-09-02T07:24:05.292 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ skipping: [smithi175.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930259316736, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227114091, 'block_used': 12981017, 'inode_total': 61054976, 'inode_available': 60985860, 'inode_used': 69116, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227114091, "block_size": 4096, "block_total": 240095108, "block_used": 12981017, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985860, "inode_total": 61054976, "inode_used": 69116, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930259316736, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:05.316 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:05 +0000 (0:00:00.031) 0:04:01.826 **** 2021-09-02T07:24:05.334 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930259316736, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227114091, 'block_used': 12981017, 'inode_total': 61054976, 'inode_available': 60985860, 'inode_used': 69116, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227114091, "block_size": 4096, "block_total": 240095108, "block_used": 12981017, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985860, "inode_total": 61054976, "inode_used": 69116, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930259316736, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:05.360 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:05 +0000 (0:00:00.041) 0:04:01.868 **** 2021-09-02T07:24:05.364 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-09-02T07:24:05.366 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:05.389 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:05 +0000 (0:00:00.031) 0:04:01.899 **** 2021-09-02T07:24:05.396 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:05.422 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:05 +0000 (0:00:00.032) 0:04:01.932 **** 2021-09-02T07:24:05.429 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-02T07:24:05.431 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:05.453 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:05 +0000 (0:00:00.031) 0:04:01.963 **** 2021-09-02T07:24:05.461 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:05.486 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:05 +0000 (0:00:00.032) 0:04:01.996 **** 2021-09-02T07:24:05.493 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:05.518 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:05 +0000 (0:00:00.032) 0:04:02.028 **** 2021-09-02T07:24:05.525 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:05.550 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:05 +0000 (0:00:00.031) 0:04:02.060 **** 2021-09-02T07:24:05.557 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:05.581 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:05 +0000 (0:00:00.031) 0:04:02.091 **** 2021-09-02T07:24:05.583 INFO:teuthology.task.ansible.out: 2021-09-02T07:24:05.588 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:05.612 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:05 +0000 (0:00:00.030) 0:04:02.122 **** 2021-09-02T07:24:05.619 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:05.650 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:05 +0000 (0:00:00.038) 0:04:02.160 **** 2021-09-02T07:24:05.657 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:05.684 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:05 +0000 (0:00:00.034) 0:04:02.194 **** 2021-09-02T07:24:05.691 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-09-02T07:24:05.693 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:05.715 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:05 +0000 (0:00:00.031) 0:04:02.226 **** 2021-09-02T07:24:05.723 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:05.747 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:05 +0000 (0:00:00.031) 0:04:02.257 **** 2021-09-02T07:24:06.283 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-09-02T07:24:06.302 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:06 +0000 (0:00:00.553) 0:04:02.811 **** 2021-09-02T07:24:06.323 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:06 +0000 (0:00:00.022) 0:04:02.833 **** 2021-09-02T07:24:06.781 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-09-02T07:24:06.783 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-02T07:24:07.120 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-02T07:24:07.140 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-02T07:24:07.463 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-02T07:24:07.517 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-02T07:24:07.811 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-02T07:24:07.875 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-02T07:24:08.206 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-02T07:24:08.230 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:08 +0000 (0:00:01.906) 0:04:04.739 **** 2021-09-02T07:24:08.240 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-02T07:24:08.271 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:08 +0000 (0:00:00.039) 0:04:04.779 **** 2021-09-02T07:24:08.454 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-09-02T07:24:08.475 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:08 +0000 (0:00:00.205) 0:04:04.985 **** 2021-09-02T07:24:08.509 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-09-02T07:24:08.559 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:08 +0000 (0:00:00.077) 0:04:05.062 **** 2021-09-02T07:24:08.689 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-09-02T07:24:08.691 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-02T07:24:08.764 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-02T07:24:08.863 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-02T07:24:08.938 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-02T07:24:09.041 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-02T07:24:09.116 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-02T07:24:09.217 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-02T07:24:09.227 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:09.253 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:09 +0000 (0:00:00.701) 0:04:05.763 **** 2021-09-02T07:24:09.292 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-09-02T07:24:09.294 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi175.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-02T07:24:09.315 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:09 +0000 (0:00:00.061) 0:04:05.825 **** 2021-09-02T07:24:09.321 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:09.350 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:09 +0000 (0:00:00.034) 0:04:05.859 **** 2021-09-02T07:24:09.360 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:09.384 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:09 +0000 (0:00:00.034) 0:04:05.894 **** 2021-09-02T07:24:09.391 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-09-02T07:24:09.393 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:09.419 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:09 +0000 (0:00:00.032) 0:04:05.927 **** 2021-09-02T07:24:09.458 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:09.473 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:09 +0000 (0:00:00.056) 0:04:05.983 **** 2021-09-02T07:24:09.577 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-02T07:24:09.602 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:09 +0000 (0:00:00.128) 0:04:06.111 **** 2021-09-02T07:24:09.639 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi197.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-09-02T07:24:09.663 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:09 +0000 (0:00:00.062) 0:04:06.174 **** 2021-09-02T07:24:09.681 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-02T07:24:09.706 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:09 +0000 (0:00:00.042) 0:04:06.216 **** 2021-09-02T07:24:09.749 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-09-02T07:24:09.751 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-09-02T07:24:09.773 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:09 +0000 (0:00:00.066) 0:04:06.283 **** 2021-09-02T07:24:10.039 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-09-02T07:24:10.041 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true} 2021-09-02T07:24:10.067 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:10 +0000 (0:00:00.292) 0:04:06.575 **** 2021-09-02T07:24:10.072 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"changed": true} 2021-09-02T07:24:10.101 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:10 +0000 (0:00:00.035) 0:04:06.610 **** 2021-09-02T07:24:10.457 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-09-02T07:24:10.459 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-09-02T07:24:10.482 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:10 +0000 (0:00:00.381) 0:04:06.992 **** 2021-09-02T07:24:10.490 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-09-02T07:24:10.512 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:10 +0000 (0:00:00.030) 0:04:07.022 **** 2021-09-02T07:24:10.520 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:10.543 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:10 +0000 (0:00:00.030) 0:04:07.053 **** 2021-09-02T07:24:10.550 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:10.573 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:10 +0000 (0:00:00.030) 0:04:07.083 **** 2021-09-02T07:24:10.579 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-09-02T07:24:10.581 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:10.608 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:10 +0000 (0:00:00.030) 0:04:07.114 **** 2021-09-02T07:24:10.611 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:10.633 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:10 +0000 (0:00:00.029) 0:04:07.143 **** 2021-09-02T07:24:10.645 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:10.670 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:10 +0000 (0:00:00.037) 0:04:07.180 **** 2021-09-02T07:24:10.678 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:10.702 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:10 +0000 (0:00:00.029) 0:04:07.210 **** 2021-09-02T07:24:10.706 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:10.730 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:10 +0000 (0:00:00.029) 0:04:07.240 **** 2021-09-02T07:24:10.737 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:10.761 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:10 +0000 (0:00:00.030) 0:04:07.270 **** 2021-09-02T07:24:10.767 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-09-02T07:24:10.769 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:10.793 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:10 +0000 (0:00:00.032) 0:04:07.302 **** 2021-09-02T07:24:10.801 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:10.841 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:10 +0000 (0:00:00.034) 0:04:07.337 **** TASK [testnode : Add CPAN config directory for the root user.] ***************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:10.867 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:10 +0000 (0:00:00.040) 0:04:07.377 **** 2021-09-02T07:24:10.875 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:10.907 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:10 +0000 (0:00:00.040) 0:04:07.418 **** 2021-09-02T07:24:10.913 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-09-02T07:24:10.915 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:10.939 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:10 +0000 (0:00:00.030) 0:04:07.448 **** 2021-09-02T07:24:10.944 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:10.969 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:10 +0000 (0:00:00.030) 0:04:07.479 **** 2021-09-02T07:24:10.980 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:11.005 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:10 +0000 (0:00:00.035) 0:04:07.515 **** 2021-09-02T07:24:11.013 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:11.034 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:11 +0000 (0:00:00.029) 0:04:07.544 **** 2021-09-02T07:24:11.043 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:11.067 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:11 +0000 (0:00:00.032) 0:04:07.577 **** 2021-09-02T07:24:11.074 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:11.097 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:11 +0000 (0:00:00.030) 0:04:07.608 **** 2021-09-02T07:24:11.104 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:11.131 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:11 +0000 (0:00:00.034) 0:04:07.642 **** 2021-09-02T07:24:11.153 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:11.175 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:11 +0000 (0:00:00.040) 0:04:07.683 **** 2021-09-02T07:24:11.179 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-09-02T07:24:11.180 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:11.203 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:11 +0000 (0:00:00.030) 0:04:07.713 **** 2021-09-02T07:24:11.212 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:11.239 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:11 +0000 (0:00:00.035) 0:04:07.748 **** 2021-09-02T07:24:11.246 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:11.271 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:11 +0000 (0:00:00.032) 0:04:07.781 **** 2021-09-02T07:24:11.281 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:11.307 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:11 +0000 (0:00:00.035) 0:04:07.817 **** 2021-09-02T07:24:11.313 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-09-02T07:24:11.314 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:11.339 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:11 +0000 (0:00:00.032) 0:04:07.849 **** 2021-09-02T07:24:11.347 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:11.370 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:11 +0000 (0:00:00.030) 0:04:07.880 **** 2021-09-02T07:24:11.391 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-09-02T07:24:11.393 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:11.415 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:11 +0000 (0:00:00.043) 0:04:07.923 **** 2021-09-02T07:24:11.420 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:11.444 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:11 +0000 (0:00:00.030) 0:04:07.954 **** 2021-09-02T07:24:11.451 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-09-02T07:24:11.453 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:11.480 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:11 +0000 (0:00:00.036) 0:04:07.990 **** 2021-09-02T07:24:11.488 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:11.511 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:11 +0000 (0:00:00.031) 0:04:08.022 **** 2021-09-02T07:24:11.515 INFO:teuthology.task.ansible.out: 2021-09-02T07:24:11.969 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-09-02T07:24:11.971 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-09-02T07:24:11.995 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:11 +0000 (0:00:00.483) 0:04:08.505 **** 2021-09-02T07:24:12.034 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-09-02T07:24:12.050 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:12 +0000 (0:00:00.054) 0:04:08.560 **** 2021-09-02T07:24:12.329 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-09-02T07:24:12.330 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveState": "active", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "879", "ExecMainStartTimestamp": "Thu 2021-09-02 07:19:45 UTC", "ExecMainStartTimestampMonotonic": "12459401", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSI 2021-09-02T07:24:12.332 INFO:teuthology.task.ansible.out:ZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "879", "MemoryAccounting": "yes", "MemoryCurrent": "1814528", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203592", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Type": "forking", "UID": "[not set]", "UMask": "0022", "WatchdogTimestamp": "Thu 2021-09-02 0 2021-09-02T07:24:12.333 INFO:teuthology.task.ansible.out:7:19:45 UTC", "WatchdogTimestampMonotonic": "12459412", "WatchdogUSec": "0"}} 2021-09-02T07:24:12.362 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:12 +0000 (0:00:00.311) 0:04:08.871 **** 2021-09-02T07:24:12.385 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveState": "active", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "789", "ExecMainStartTimestamp": "Thu 2021-09-02 07:19:48 UTC", "ExecMainStartTimestampMonotonic": "11707829", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSI 2021-09-02T07:24:12.387 INFO:teuthology.task.ansible.out:ZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127246", "LimitNPROCSoft": "127246", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127246", "LimitSIGPENDINGSoft": "127246", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "789", "MemoryAccounting": "yes", "MemoryCurrent": "2064384", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203593", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Type": "forking", "UID": "[not set]", "UMask": "0022", "WatchdogTimestamp": "Thu 2021-09-02 0 2021-09-02T07:24:12.389 INFO:teuthology.task.ansible.out:7:19:48 UTC", "WatchdogTimestampMonotonic": "11707840", "WatchdogUSec": "0"}} 2021-09-02T07:24:12.409 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:12 +0000 (0:00:00.048) 0:04:08.920 **** 2021-09-02T07:24:12.417 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-02T07:24:12.419 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-09-02T07:24:12.442 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:12 +0000 (0:00:00.032) 0:04:08.952 **** 2021-09-02T07:24:12.451 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-09-02T07:24:12.479 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:12 +0000 (0:00:00.036) 0:04:08.989 **** 2021-09-02T07:24:12.487 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-02T07:24:12.488 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-09-02T07:24:12.516 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:12 +0000 (0:00:00.037) 0:04:09.026 **** 2021-09-02T07:24:12.527 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-09-02T07:24:12.551 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:12 +0000 (0:00:00.034) 0:04:09.061 **** 2021-09-02T07:24:12.560 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:12.592 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:12 +0000 (0:00:00.038) 0:04:09.100 **** 2021-09-02T07:24:12.599 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:12.625 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:12 +0000 (0:00:00.034) 0:04:09.135 **** 2021-09-02T07:24:12.808 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 02 September 2021 07:24:12 +0000 (0:00:00.048) 0:04:09.183 **** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 02 September 2021 07:24:12 +0000 (0:00:00.044) 0:04:09.227 **** TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 02 September 2021 07:24:12 +0000 (0:00:00.032) 0:04:09.259 **** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 02 September 2021 07:24:12 +0000 (0:00:00.034) 0:04:09.294 **** 2021-09-02T07:24:12.989 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-09-02T07:24:12.995 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "gid": 1254, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-09-02T07:24:13.019 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:13 +0000 (0:00:00.234) 0:04:09.529 **** 2021-09-02T07:24:13.043 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "gid": 1254, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-09-02T07:24:13.068 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:13 +0000 (0:00:00.048) 0:04:09.578 **** 2021-09-02T07:24:13.542 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1254, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-09-02T07:24:13.565 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:13 +0000 (0:00:00.497) 0:04:10.075 **** 2021-09-02T07:24:13.574 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1254, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-09-02T07:24:13.610 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:13 +0000 (0:00:00.044) 0:04:10.120 **** 2021-09-02T07:24:14.650 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-09-02T07:24:14.652 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-09-02T07:24:14.675 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:14 +0000 (0:00:01.064) 0:04:11.184 **** 2021-09-02T07:24:14.702 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:14.727 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:14 +0000 (0:00:00.053) 0:04:11.238 **** 2021-09-02T07:24:14.735 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-09-02T07:24:14.736 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-09-02T07:24:14.759 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:14 +0000 (0:00:00.031) 0:04:11.270 **** 2021-09-02T07:24:14.779 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ ok: [smithi175.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-09-02T07:24:14.801 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:14 +0000 (0:00:00.039) 0:04:11.309 **** 2021-09-02T07:24:14.810 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:14.830 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:14 +0000 (0:00:00.030) 0:04:11.340 **** 2021-09-02T07:24:14.871 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ ok: [smithi197.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-09-02T07:24:14.896 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:14 +0000 (0:00:00.063) 0:04:11.403 **** 2021-09-02T07:24:14.988 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-09-02T07:24:15.016 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:15 +0000 (0:00:00.117) 0:04:11.521 **** 2021-09-02T07:24:15.040 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-09-02T07:24:15.044 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:15.063 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:15 +0000 (0:00:00.052) 0:04:11.573 **** 2021-09-02T07:24:15.097 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi175.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-09-02T07:24:15.118 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:15 +0000 (0:00:00.055) 0:04:11.628 **** 2021-09-02T07:24:15.140 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:15 +0000 (0:00:00.020) 0:04:11.649 **** 2021-09-02T07:24:15.157 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:15.188 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:15 +0000 (0:00:00.048) 0:04:11.697 **** 2021-09-02T07:24:15.196 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-09-02T07:24:15.200 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:15.225 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:15 +0000 (0:00:00.036) 0:04:11.734 **** 2021-09-02T07:24:15.279 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi197.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-09-02T07:24:15.305 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:15 +0000 (0:00:00.075) 0:04:11.809 **** 2021-09-02T07:24:15.341 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:15.363 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:15 +0000 (0:00:00.063) 0:04:11.873 **** 2021-09-02T07:24:15.391 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-09-02T07:24:15.392 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-09-02T07:24:15.415 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:15 +0000 (0:00:00.051) 0:04:11.925 **** 2021-09-02T07:24:15.560 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-09-02T07:24:15.585 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:15 +0000 (0:00:00.170) 0:04:12.095 **** 2021-09-02T07:24:15.636 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-09-02T07:24:15.638 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-09-02T07:24:15.670 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:15 +0000 (0:00:00.082) 0:04:12.178 **** 2021-09-02T07:24:15.700 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-09-02T07:24:15.702 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:15.730 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:15 +0000 (0:00:00.061) 0:04:12.240 **** 2021-09-02T07:24:15.785 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi197.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi175.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-09-02T07:24:15.839 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:15 +0000 (0:00:00.086) 0:04:12.327 **** skipping: [smithi175.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-09-02T07:24:15.867 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:15 +0000 (0:00:00.041) 0:04:12.368 **** 2021-09-02T07:24:15.880 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:15.904 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:15 +0000 (0:00:00.045) 0:04:12.414 **** 2021-09-02T07:24:15.924 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi175.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:15.925 INFO:teuthology.task.ansible.out: 2021-09-02T07:24:15.938 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:15.954 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi197.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi175.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi175.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:15.993 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:15 +0000 (0:00:00.088) 0:04:12.503 **** 2021-09-02T07:24:16.006 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-09-02T07:24:16.008 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-09-02T07:24:16.010 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-09-02T07:24:16.047 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:16 +0000 (0:00:00.054) 0:04:12.557 **** 2021-09-02T07:24:16.058 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-02T07:24:16.060 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:16.103 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:16 +0000 (0:00:00.052) 0:04:12.609 **** 2021-09-02T07:24:16.111 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-09-02T07:24:16.121 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:16.126 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi197.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:16.160 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:16 +0000 (0:00:00.060) 0:04:12.670 **** 2021-09-02T07:24:16.178 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:16.195 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:16 +0000 (0:00:00.035) 0:04:12.705 **** 2021-09-02T07:24:16.203 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-02T07:24:16.205 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:16.244 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:16 +0000 (0:00:00.049) 0:04:12.754 **** 2021-09-02T07:24:16.254 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-09-02T07:24:16.256 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:16.289 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:16 +0000 (0:00:00.043) 0:04:12.798 **** 2021-09-02T07:24:16.300 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:16.324 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:16 +0000 (0:00:00.034) 0:04:12.833 **** 2021-09-02T07:24:16.332 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-09-02T07:24:16.336 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:16.371 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:16 +0000 (0:00:00.045) 0:04:12.878 **** 2021-09-02T07:24:16.390 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:16.427 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:16 +0000 (0:00:00.051) 0:04:12.929 **** TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-09-02T07:24:16.429 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:16.459 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:16 +0000 (0:00:00.039) 0:04:12.969 **** 2021-09-02T07:24:16.469 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:16.498 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:16 +0000 (0:00:00.039) 0:04:13.008 **** 2021-09-02T07:24:16.506 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-09-02T07:24:16.514 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:16.542 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:16 +0000 (0:00:00.039) 0:04:13.047 **** 2021-09-02T07:24:16.545 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-09-02T07:24:16.547 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:16.575 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:16 +0000 (0:00:00.036) 0:04:13.084 **** 2021-09-02T07:24:16.582 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-09-02T07:24:16.584 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:16.610 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:16 +0000 (0:00:00.036) 0:04:13.120 **** 2021-09-02T07:24:16.627 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:16.653 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:16 +0000 (0:00:00.042) 0:04:13.162 **** 2021-09-02T07:24:16.660 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-09-02T07:24:16.662 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:16.695 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:16 +0000 (0:00:00.042) 0:04:13.205 **** 2021-09-02T07:24:16.704 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:16.732 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:16 +0000 (0:00:00.037) 0:04:13.242 **** 2021-09-02T07:24:16.772 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-09-02T07:24:16.773 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:16.793 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:16 +0000 (0:00:00.061) 0:04:13.304 **** 2021-09-02T07:24:16.919 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi175.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005090", "end": "2021-09-02 07:24:16.880672", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-09-02 07:24:16.875582", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:17.023 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005261", "end": "2021-09-02 07:24:16.985878", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-09-02 07:24:16.980617", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:17.087 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002924", "end": "2021-09-02 07:24:17.057714", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-09-02 07:24:17.054790", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:17.259 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002642", "end": "2021-09-02 07:24:17.226983", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-09-02 07:24:17.224341", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:17.421 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003277", "end": "2021-09-02 07:24:17.394631", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-09-02 07:24:17.391354", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:17.443 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:17 +0000 (0:00:00.649) 0:04:13.953 **** 2021-09-02T07:24:17.470 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-09-02T07:24:17.494 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:17 +0000 (0:00:00.051) 0:04:14.004 **** 2021-09-02T07:24:17.524 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:18.230 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi197.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:01.004054", "end": "2021-09-02 07:24:18.183593", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-09-02 07:24:17.179539", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:18.418 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002865", "end": "2021-09-02 07:24:18.387271", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-09-02 07:24:18.384406", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:18.624 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003118", "end": "2021-09-02 07:24:18.597461", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-09-02 07:24:18.594343", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:18.647 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:18 +0000 (0:00:01.152) 0:04:15.157 **** 2021-09-02T07:24:18.686 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-09-02T07:24:18.716 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:18 +0000 (0:00:00.065) 0:04:15.222 **** 2021-09-02T07:24:18.742 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:18.817 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-09-02T07:24:18.842 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:18 +0000 (0:00:00.125) 0:04:15.347 **** 2021-09-02T07:24:18.869 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:18 +0000 (0:00:00.022) 0:04:15.370 **** 2021-09-02T07:24:19.518 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi175.front.sepia.ceph.com] 2021-09-02T07:24:19.562 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:19 +0000 (0:00:00.703) 0:04:16.073 **** 2021-09-02T07:24:19.592 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:19.614 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:19 +0000 (0:00:00.051) 0:04:16.124 **** 2021-09-02T07:24:19.658 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi175.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-09-02T07:24:19.682 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:19 +0000 (0:00:00.065) 0:04:16.190 **** 2021-09-02T07:24:19.734 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** ok: [smithi175.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-09-02T07:24:19.754 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:19 +0000 (0:00:00.073) 0:04:16.264 **** 2021-09-02T07:24:21.565 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi197.front.sepia.ceph.com] 2021-09-02T07:24:21.614 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:21 +0000 (0:00:01.859) 0:04:18.123 **** 2021-09-02T07:24:21.653 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-02T07:24:21.655 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:21.677 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:21 +0000 (0:00:00.064) 0:04:18.188 **** 2021-09-02T07:24:21.718 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi197.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-09-02T07:24:21.740 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:21 +0000 (0:00:00.061) 0:04:18.250 **** 2021-09-02T07:24:21.788 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-09-02T07:24:21.790 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-09-02T07:24:21.812 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:21 +0000 (0:00:00.072) 0:04:18.322 **** 2021-09-02T07:24:25.179 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-09-02T07:24:25.181 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: podman-3.2.3-0.10.module_el8.4.0+886+c9a8d9ad.x86_64", "Installed: podman-docker-3.2.3-0.10.module_el8.4.0+886+c9a8d9ad.noarch", "Installed: podman-catatonit-3.2.3-0.10.module_el8.4.0+886+c9a8d9ad.x86_64", "Removed: podman-3.0.1-7.module_el8.4.0+830+8027e1c4.x86_64", "Removed: podman-catatonit-3.0.1-7.module_el8.4.0+830+8027e1c4.x86_64", "Removed: podman-docker-3.0.1-7.module_el8.4.0+830+8027e1c4.noarch"]} 2021-09-02T07:24:25.203 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:25 +0000 (0:00:03.391) 0:04:21.713 **** 2021-09-02T07:24:25.230 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-02T07:24:25.232 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-09-02T07:24:25.252 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:25 +0000 (0:00:00.049) 0:04:21.762 **** 2021-09-02T07:24:25.281 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-02T07:24:25.283 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:25.304 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:25 +0000 (0:00:00.051) 0:04:21.814 **** 2021-09-02T07:24:25.559 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* ok: [smithi175.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-02T07:24:25.752 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-02T07:24:25.776 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:25 +0000 (0:00:00.471) 0:04:22.286 **** 2021-09-02T07:24:26.262 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-09-02T07:24:26.263 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-09-02T07:24:26.705 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-09-02T07:24:26.731 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:26 +0000 (0:00:00.952) 0:04:23.239 **** 2021-09-02T07:24:27.236 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-09-02T07:24:27.238 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: podman-3.2.3-0.10.module_el8.4.0+886+c9a8d9ad.x86_64", "Installed: podman-docker-3.2.3-0.10.module_el8.4.0+886+c9a8d9ad.noarch", "Installed: podman-catatonit-3.2.3-0.10.module_el8.4.0+886+c9a8d9ad.x86_64", "Removed: podman-3.0.1-7.module_el8.4.0+830+8027e1c4.x86_64", "Removed: podman-catatonit-3.0.1-7.module_el8.4.0+830+8027e1c4.x86_64", "Removed: podman-docker-3.0.1-7.module_el8.4.0+830+8027e1c4.noarch"]} 2021-09-02T07:24:27.262 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:27 +0000 (0:00:00.533) 0:04:23.772 **** 2021-09-02T07:24:27.300 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-02T07:24:27.302 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-09-02T07:24:27.323 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:27 +0000 (0:00:00.060) 0:04:23.833 **** 2021-09-02T07:24:27.352 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:27.374 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:27 +0000 (0:00:00.051) 0:04:23.885 **** 2021-09-02T07:24:27.610 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-09-02T07:24:27.611 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-02T07:24:27.823 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-02T07:24:27.847 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:27 +0000 (0:00:00.472) 0:04:24.357 **** 2021-09-02T07:24:28.270 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** 2021-09-02T07:24:28.271 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-02T07:24:28.293 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:28 +0000 (0:00:00.445) 0:04:24.802 **** 2021-09-02T07:24:28.332 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-09-02T07:24:28.334 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-09-02T07:24:28.768 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-09-02T07:24:28.801 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:28 +0000 (0:00:00.508) 0:04:25.311 **** 2021-09-02T07:24:30.372 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** 2021-09-02T07:24:30.374 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-02T07:24:30.393 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:30 +0000 (0:00:01.592) 0:04:26.904 **** 2021-09-02T07:24:30.728 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** ok: [smithi175.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-7txlm8as-build\nRequirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nRequirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nInstalling collected packages: registries-conf-ctl\n Found existing installation: registries-conf-ctl 1.0\n Uninstalling registries-conf-ctl-1.0:\n Successfully uninstalled registries-conf-ctl-1.0\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed registries-conf-ctl-1.0\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-7txlm8as-build", "Requirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Requirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Installing collected packages: registries-conf-ctl", " Found existing installation: registries-conf-ctl 1.0", " Uninstalling registries-conf-ctl-1.0 2021-09-02T07:24:30.730 INFO:teuthology.task.ansible.out::", " Successfully uninstalled registries-conf-ctl-1.0", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed registries-conf-ctl-1.0"], "version": null, "virtualenv": null} 2021-09-02T07:24:30.757 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:30 +0000 (0:00:00.363) 0:04:27.268 **** 2021-09-02T07:24:30.799 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:30.830 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:30 +0000 (0:00:00.072) 0:04:27.340 **** 2021-09-02T07:24:30.869 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-09-02T07:24:30.871 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:30.893 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:30 +0000 (0:00:00.063) 0:04:27.403 **** 2021-09-02T07:24:31.249 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-09-02T07:24:31.250 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.156101", "end": "2021-09-02 07:24:31.229858", "rc": 0, "start": "2021-09-02 07:24:31.073757", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:31.272 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:31 +0000 (0:00:00.378) 0:04:27.782 **** 2021-09-02T07:24:31.300 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-09-02T07:24:31.301 INFO:teuthology.task.ansible.out:skipping: [smithi175.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:32.868 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** ok: [smithi197.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-w7clzyc4-build\nRequirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nRequirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nInstalling collected packages: registries-conf-ctl\n Found existing installation: registries-conf-ctl 1.0\n Uninstalling registries-conf-ctl-1.0:\n Successfully uninstalled registries-conf-ctl-1.0\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed registries-conf-ctl-1.0\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-w7clzyc4-build", "Requirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Requirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Installing collected packages: registries-conf-ctl", " Found existing installation: registries-conf-ctl 1.0", " Uninstalling registries-conf-ctl-1.0 2021-09-02T07:24:32.870 INFO:teuthology.task.ansible.out::", " Successfully uninstalled registries-conf-ctl-1.0", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed registries-conf-ctl-1.0"], "version": null, "virtualenv": null} 2021-09-02T07:24:32.889 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:32 +0000 (0:00:01.617) 0:04:29.399 **** 2021-09-02T07:24:32.916 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:32.938 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:32 +0000 (0:00:00.048) 0:04:29.448 **** 2021-09-02T07:24:32.975 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-09-02T07:24:32.977 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:33.000 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:32 +0000 (0:00:00.062) 0:04:29.510 **** 2021-09-02T07:24:33.365 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-09-02T07:24:33.368 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.177021", "end": "2021-09-02 07:24:33.349059", "rc": 0, "start": "2021-09-02 07:24:33.172038", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-02T07:24:33.388 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:33 +0000 (0:00:00.387) 0:04:29.898 **** 2021-09-02T07:24:33.425 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-09-02T07:24:33.427 INFO:teuthology.task.ansible.out:skipping: [smithi197.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-02T07:24:33.496 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-09-02T07:24:33.497 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-09-02T07:24:33.499 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-09-02T07:24:33.501 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-09-02T07:24:33.513 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-09-02T07:24:33.514 INFO:teuthology.task.ansible.out:Thursday 02 September 2021 07:24:33 +0000 (0:00:00.125) 0:04:30.023 **** 2021-09-02T07:24:34.171 INFO:teuthology.task.ansible.out:ok: [smithi197.front.sepia.ceph.com] 2021-09-02T07:24:35.175 INFO:teuthology.task.ansible.out:ok: [smithi175.front.sepia.ceph.com] 2021-09-02T07:24:35.198 INFO:teuthology.task.ansible.out: TASK [Touch /ceph-qa-ready] **************************************************** Thursday 02 September 2021 07:24:35 +0000 (0:00:01.685) 0:04:31.709 **** 2021-09-02T07:24:35.394 INFO:teuthology.task.ansible.out:changed: [smithi175.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-09-02T07:24:35.436 INFO:teuthology.task.ansible.out:changed: [smithi197.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-09-02T07:24:35.438 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* 2021-09-02T07:24:35.440 INFO:teuthology.task.ansible.out:smithi175.front.sepia.ceph.com : ok=132 changed=39 unreachable=0 failed=0 skipped=188 rescued=0 ignored=3 smithi197.front.sepia.ceph.com : ok=132 changed=39 unreachable=0 failed=0 skipped=188 rescued=0 ignored=3 Thursday 02 September 2021 07:24:35 +0000 (0:00:00.240) 0:04:31.949 **** =============================================================================== common : nrpe - Ensure SELinux policy is up to date -------------------- 37.10s Create all admin users with sudo access. ------------------------------- 35.18s users : Update authorized_keys using the keys repo --------------------- 32.88s common : nrpe - Remove SELinux policy package -------------------------- 14.98s common : nrpe - Load SELinux policy package ---------------------------- 14.95s Remove revoked users ---------------------------------------------------- 9.39s testnode : Zap all non-root disks --------------------------------------- 7.70s testnode : Install packages --------------------------------------------- 6.99s testnode : Ensure ceph packages are not present. ------------------------ 4.30s common : Install Kerberos Packages (RedHat) ----------------------------- 4.02s container-host : Install container packages ----------------------------- 3.39s testnode : Rebuild rpmdb ------------------------------------------------ 2.49s testnode : Ensure ceph dependency packages are not present. ------------- 2.37s testnode : Create logical volume(s) ------------------------------------- 1.91s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.86s testnode : Configure local mirrorlists ---------------------------------- 1.83s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 1.80s Gathering Facts --------------------------------------------------------- 1.69s users : Update authorized_keys for each user with literal keys ---------- 1.61s container-host : Ensure git is installed -------- 2021-09-02T07:24:35.442 INFO:teuthology.task.ansible.out:------------------------ 1.59s 2021-09-02T07:24:35.857 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi175.front.sepia.ceph.com'), Remote(name='ubuntu@smithi197.front.sepia.ceph.com')] 2021-09-02T07:24:35.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-02T07:24:35.860 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-02T07:24:35.928 DEBUG:teuthology.orchestra.run.smithi175:> true 2021-09-02T07:24:36.015 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-02T07:24:36.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi197.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-02T07:24:36.086 DEBUG:teuthology.orchestra.run.smithi197:> true 2021-09-02T07:24:36.171 INFO:teuthology.run_tasks:Running task clock... 2021-09-02T07:24:36.183 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-09-02T07:24:36.183 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-09-02T07:24:36.185 DEBUG:teuthology.orchestra.run.smithi175:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-09-02T07:24:36.218 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-09-02T07:24:36.233 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-09-02T07:24:36.266 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: ntpd: command not found 2021-09-02T07:24:36.276 INFO:teuthology.orchestra.run.smithi175.stdout:506 Cannot talk to daemon 2021-09-02T07:24:36.289 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-09-02T07:24:36.302 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-09-02T07:24:36.363 INFO:teuthology.orchestra.run.smithi175.stderr:bash: ntpq: command not found 2021-09-02T07:24:36.365 INFO:teuthology.orchestra.run.smithi175.stdout:210 Number of sources = 4 2021-09-02T07:24:36.366 INFO:teuthology.orchestra.run.smithi175.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-09-02T07:24:36.366 INFO:teuthology.orchestra.run.smithi175.stdout:=============================================================================== 2021-09-02T07:24:36.366 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-02T07:24:36.367 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-02T07:24:36.367 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-02T07:24:36.367 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-02T07:24:36.367 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-09-02T07:24:36.368 DEBUG:teuthology.orchestra.run.smithi197:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-09-02T07:24:36.400 INFO:teuthology.orchestra.run.smithi197.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-09-02T07:24:36.415 INFO:teuthology.orchestra.run.smithi197.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-09-02T07:24:36.449 INFO:teuthology.orchestra.run.smithi197.stderr:sudo: ntpd: command not found 2021-09-02T07:24:36.458 INFO:teuthology.orchestra.run.smithi197.stdout:506 Cannot talk to daemon 2021-09-02T07:24:36.472 INFO:teuthology.orchestra.run.smithi197.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-09-02T07:24:36.486 INFO:teuthology.orchestra.run.smithi197.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-09-02T07:24:36.559 INFO:teuthology.orchestra.run.smithi197.stderr:bash: ntpq: command not found 2021-09-02T07:24:36.563 INFO:teuthology.orchestra.run.smithi197.stdout:210 Number of sources = 4 2021-09-02T07:24:36.563 INFO:teuthology.orchestra.run.smithi197.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-09-02T07:24:36.563 INFO:teuthology.orchestra.run.smithi197.stdout:=============================================================================== 2021-09-02T07:24:36.563 INFO:teuthology.orchestra.run.smithi197.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-02T07:24:36.564 INFO:teuthology.orchestra.run.smithi197.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-02T07:24:36.564 INFO:teuthology.orchestra.run.smithi197.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-02T07:24:36.564 INFO:teuthology.orchestra.run.smithi197.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-02T07:24:36.564 INFO:teuthology.run_tasks:Running task install... 2021-09-02T07:24:36.575 DEBUG:teuthology.task.install:project ceph 2021-09-02T07:24:36.575 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '9366f375ac31713c35e559edbd160565b784db1d'}} 2021-09-02T07:24:36.576 DEBUG:teuthology.task.install:config {'sha1': '9366f375ac31713c35e559edbd160565b784db1d'} 2021-09-02T07:24:36.576 INFO:teuthology.task.install:Using flavor: basic 2021-09-02T07:24:36.583 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-09-02T07:24:36.583 INFO:teuthology.task.install:extra packages: [] 2021-09-02T07:24:36.584 DEBUG:teuthology.orchestra.run.smithi175:> sudo lsb_release -is 2021-09-02T07:24:36.617 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS 2021-09-02T07:24:36.618 DEBUG:teuthology.misc:System to be installed: CentOS 2021-09-02T07:24:36.619 DEBUG:teuthology.orchestra.run.smithi197:> sudo lsb_release -is 2021-09-02T07:24:36.620 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9366f375ac31713c35e559edbd160565b784db1d', 'tag': None, 'wait_for_package': False} 2021-09-02T07:24:36.621 DEBUG:teuthology.orchestra.run.smithi175:> sudo lsb_release -is 2021-09-02T07:24:36.653 INFO:teuthology.orchestra.run.smithi197.stdout:CentOS 2021-09-02T07:24:36.654 DEBUG:teuthology.misc:System to be installed: CentOS 2021-09-02T07:24:36.655 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9366f375ac31713c35e559edbd160565b784db1d', 'tag': None, 'wait_for_package': False} 2021-09-02T07:24:36.656 DEBUG:teuthology.orchestra.run.smithi197:> sudo lsb_release -is 2021-09-02T07:24:36.689 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS 2021-09-02T07:24:36.690 DEBUG:teuthology.misc:System to be installed: CentOS 2021-09-02T07:24:36.691 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9366f375ac31713c35e559edbd160565b784db1d 2021-09-02T07:24:36.726 INFO:teuthology.orchestra.run.smithi197.stdout:CentOS 2021-09-02T07:24:36.727 DEBUG:teuthology.misc:System to be installed: CentOS 2021-09-02T07:24:36.727 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9366f375ac31713c35e559edbd160565b784db1d 2021-09-02T07:24:36.843 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-skoduri-dbstore-object/9366f375ac31713c35e559edbd160565b784db1d/centos/8/flavors/default/ 2021-09-02T07:24:36.844 INFO:teuthology.task.install.rpm:Package version is 17.0.0-7401.g9366f375 2021-09-02T07:24:36.876 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-skoduri-dbstore-object/9366f375ac31713c35e559edbd160565b784db1d/centos/8/flavors/default/ 2021-09-02T07:24:36.877 INFO:teuthology.task.install.rpm:Package version is 17.0.0-7401.g9366f375 2021-09-02T07:24:37.020 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-skoduri-dbstore-object/9366f375ac31713c35e559edbd160565b784db1d/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-skoduri-dbstore-object/9366f375ac31713c35e559edbd160565b784db1d/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-skoduri-dbstore-object/9366f375ac31713c35e559edbd160565b784db1d/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-09-02T07:24:37.021 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:24:37.021 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-09-02T07:24:37.046 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-09-02T07:24:37.046 DEBUG:teuthology.orchestra.run.smithi175:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9366f375ac31713c35e559edbd160565b784db1d/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-09-02T07:24:37.054 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-skoduri-dbstore-object/9366f375ac31713c35e559edbd160565b784db1d/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-skoduri-dbstore-object/9366f375ac31713c35e559edbd160565b784db1d/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-skoduri-dbstore-object/9366f375ac31713c35e559edbd160565b784db1d/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-09-02T07:24:37.054 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:24:37.054 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-09-02T07:24:37.080 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-09-02T07:24:37.081 DEBUG:teuthology.orchestra.run.smithi197:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9366f375ac31713c35e559edbd160565b784db1d/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-09-02T07:24:37.111 DEBUG:teuthology.orchestra.run.smithi175:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-09-02T07:24:37.145 DEBUG:teuthology.orchestra.run.smithi197:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-09-02T07:24:37.184 DEBUG:teuthology.orchestra.run.smithi175:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-09-02T07:24:37.218 DEBUG:teuthology.orchestra.run.smithi197:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-09-02T07:24:37.246 INFO:teuthology.orchestra.run.smithi175.stdout:check_obsoletes = 1 2021-09-02T07:24:37.248 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum clean all 2021-09-02T07:24:37.280 INFO:teuthology.orchestra.run.smithi197.stdout:check_obsoletes = 1 2021-09-02T07:24:37.282 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum clean all 2021-09-02T07:24:37.478 INFO:teuthology.orchestra.run.smithi175.stdout:84 files removed 2021-09-02T07:24:37.498 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-radosgw 2021-09-02T07:24:37.511 INFO:teuthology.orchestra.run.smithi197.stdout:84 files removed 2021-09-02T07:24:37.531 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install ceph-radosgw 2021-09-02T07:24:38.131 INFO:teuthology.orchestra.run.smithi175.stdout:ceph packages for x86_64 247 kB/s | 76 kB 00:00 2021-09-02T07:24:38.156 INFO:teuthology.orchestra.run.smithi197.stdout:ceph packages for x86_64 254 kB/s | 76 kB 00:00 2021-09-02T07:24:38.349 INFO:teuthology.orchestra.run.smithi175.stdout:ceph noarch packages 39 kB/s | 7.4 kB 00:00 2021-09-02T07:24:38.382 INFO:teuthology.orchestra.run.smithi197.stdout:ceph noarch packages 38 kB/s | 7.4 kB 00:00 2021-09-02T07:24:38.574 INFO:teuthology.orchestra.run.smithi175.stdout:ceph source packages 7.1 kB/s | 1.4 kB 00:00 2021-09-02T07:24:38.596 INFO:teuthology.orchestra.run.smithi197.stdout:ceph source packages 7.5 kB/s | 1.4 kB 00:00 2021-09-02T07:24:38.789 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS-8 - AppStream 46 MB/s | 8.8 MB 00:00 2021-09-02T07:24:38.836 INFO:teuthology.orchestra.run.smithi197.stdout:CentOS-8 - AppStream 41 MB/s | 8.8 MB 00:00 2021-09-02T07:24:40.279 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS-8 - Base 42 MB/s | 5.6 MB 00:00 2021-09-02T07:24:40.331 INFO:teuthology.orchestra.run.smithi197.stdout:CentOS-8 - Base 39 MB/s | 5.6 MB 00:00 2021-09-02T07:24:41.098 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS-8 - Extras 160 kB/s | 10 kB 00:00 2021-09-02T07:24:41.197 INFO:teuthology.orchestra.run.smithi197.stdout:CentOS-8 - Extras 155 kB/s | 10 kB 00:00 2021-09-02T07:24:44.862 INFO:teuthology.orchestra.run.smithi197.stdout:CentOS Linux 8 - AppStream 2.4 MB/s | 8.8 MB 00:03 2021-09-02T07:24:44.937 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Linux 8 - AppStream 2.3 MB/s | 8.8 MB 00:03 2021-09-02T07:24:47.004 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Linux 8 - BaseOS 8.1 MB/s | 5.6 MB 00:00 2021-09-02T07:24:47.146 INFO:teuthology.orchestra.run.smithi197.stdout:CentOS Linux 8 - BaseOS 6.4 MB/s | 5.6 MB 00:00 2021-09-02T07:24:48.096 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Linux 8 - Extras 33 kB/s | 10 kB 00:00 2021-09-02T07:24:48.197 INFO:teuthology.orchestra.run.smithi197.stdout:CentOS Linux 8 - Extras 38 kB/s | 10 kB 00:00 2021-09-02T07:24:48.238 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS-8 - PowerTools 27 MB/s | 2.3 MB 00:00 2021-09-02T07:24:48.346 INFO:teuthology.orchestra.run.smithi197.stdout:CentOS-8 - PowerTools 25 MB/s | 2.3 MB 00:00 2021-09-02T07:24:50.918 INFO:teuthology.orchestra.run.smithi175.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2021-09-02T07:24:50.931 INFO:teuthology.orchestra.run.smithi197.stdout:Copr repo for python3-asyncssh owned by ceph 1.6 kB/s | 3.5 kB 00:02 2021-09-02T07:24:51.663 INFO:teuthology.orchestra.run.smithi175.stdout:Extra Packages for Enterprise Linux 15 MB/s | 10 MB 00:00 2021-09-02T07:24:51.757 INFO:teuthology.orchestra.run.smithi197.stdout:Extra Packages for Enterprise Linux 13 MB/s | 10 MB 00:00 2021-09-02T07:24:53.794 INFO:teuthology.orchestra.run.smithi175.stdout:lab-extras 4.7 MB/s | 24 kB 00:00 2021-09-02T07:24:53.905 INFO:teuthology.orchestra.run.smithi197.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2021-09-02T07:24:55.987 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:24:55.989 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:24:55.989 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2021-09-02T07:24:55.989 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:24:55.990 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2021-09-02T07:24:55.990 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-radosgw x86_64 2:17.0.0-7401.g9366f375.el8 ceph 9.0 M 2021-09-02T07:24:55.990 INFO:teuthology.orchestra.run.smithi175.stdout:Upgrading: 2021-09-02T07:24:55.991 INFO:teuthology.orchestra.run.smithi175.stdout: librados2 x86_64 2:17.0.0-7401.g9366f375.el8 ceph 3.4 M 2021-09-02T07:24:55.991 INFO:teuthology.orchestra.run.smithi175.stdout: librbd1 x86_64 2:17.0.0-7401.g9366f375.el8 ceph 3.9 M 2021-09-02T07:24:55.991 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2021-09-02T07:24:55.991 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-base x86_64 2:17.0.0-7401.g9366f375.el8 ceph 5.2 M 2021-09-02T07:24:55.992 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-common x86_64 2:17.0.0-7401.g9366f375.el8 ceph 20 M 2021-09-02T07:24:55.992 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-selinux x86_64 2:17.0.0-7401.g9366f375.el8 ceph 24 k 2021-09-02T07:24:55.992 INFO:teuthology.orchestra.run.smithi175.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-09-02T07:24:55.992 INFO:teuthology.orchestra.run.smithi175.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-09-02T07:24:55.993 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs2 x86_64 2:17.0.0-7401.g9366f375.el8 ceph 784 k 2021-09-02T07:24:55.993 INFO:teuthology.orchestra.run.smithi175.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-09-02T07:24:55.993 INFO:teuthology.orchestra.run.smithi175.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2021-09-02T07:24:55.993 INFO:teuthology.orchestra.run.smithi175.stdout: libradosstriper1 x86_64 2:17.0.0-7401.g9366f375.el8 ceph 482 k 2021-09-02T07:24:55.994 INFO:teuthology.orchestra.run.smithi175.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2021-09-02T07:24:55.994 INFO:teuthology.orchestra.run.smithi175.stdout: librgw2 x86_64 2:17.0.0-7401.g9366f375.el8 ceph 3.8 M 2021-09-02T07:24:55.994 INFO:teuthology.orchestra.run.smithi175.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-09-02T07:24:55.994 INFO:teuthology.orchestra.run.smithi175.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-09-02T07:24:55.995 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-argparse 2021-09-02T07:24:55.995 INFO:teuthology.orchestra.run.smithi175.stdout: x86_64 2:17.0.0-7401.g9366f375.el8 ceph 45 k 2021-09-02T07:24:55.995 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-common x86_64 2:17.0.0-7401.g9366f375.el8 ceph 78 k 2021-09-02T07:24:55.995 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cephfs x86_64 2:17.0.0-7401.g9366f375.el8 ceph 178 k 2021-09-02T07:24:55.996 INFO:teuthology.orchestra.run.smithi175.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-09-02T07:24:55.996 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rados x86_64 2:17.0.0-7401.g9366f375.el8 ceph 365 k 2021-09-02T07:24:55.996 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rbd x86_64 2:17.0.0-7401.g9366f375.el8 ceph 346 k 2021-09-02T07:24:55.996 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rgw x86_64 2:17.0.0-7401.g9366f375.el8 ceph 110 k 2021-09-02T07:24:55.997 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:24:55.997 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2021-09-02T07:24:55.997 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:24:55.997 INFO:teuthology.orchestra.run.smithi175.stdout:Install 21 Packages 2021-09-02T07:24:55.997 INFO:teuthology.orchestra.run.smithi175.stdout:Upgrade 2 Packages 2021-09-02T07:24:55.998 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:24:55.999 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 49 M 2021-09-02T07:24:55.999 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2021-09-02T07:24:56.143 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:24:56.145 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:24:56.145 INFO:teuthology.orchestra.run.smithi197.stdout: Package Arch Version Repository Size 2021-09-02T07:24:56.145 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:24:56.146 INFO:teuthology.orchestra.run.smithi197.stdout:Installing: 2021-09-02T07:24:56.146 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-radosgw x86_64 2:17.0.0-7401.g9366f375.el8 ceph 9.0 M 2021-09-02T07:24:56.146 INFO:teuthology.orchestra.run.smithi197.stdout:Upgrading: 2021-09-02T07:24:56.146 INFO:teuthology.orchestra.run.smithi197.stdout: librados2 x86_64 2:17.0.0-7401.g9366f375.el8 ceph 3.4 M 2021-09-02T07:24:56.147 INFO:teuthology.orchestra.run.smithi197.stdout: librbd1 x86_64 2:17.0.0-7401.g9366f375.el8 ceph 3.9 M 2021-09-02T07:24:56.147 INFO:teuthology.orchestra.run.smithi197.stdout:Installing dependencies: 2021-09-02T07:24:56.147 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-base x86_64 2:17.0.0-7401.g9366f375.el8 ceph 5.2 M 2021-09-02T07:24:56.148 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-common x86_64 2:17.0.0-7401.g9366f375.el8 ceph 20 M 2021-09-02T07:24:56.148 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-selinux x86_64 2:17.0.0-7401.g9366f375.el8 ceph 24 k 2021-09-02T07:24:56.148 INFO:teuthology.orchestra.run.smithi197.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-09-02T07:24:56.148 INFO:teuthology.orchestra.run.smithi197.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-09-02T07:24:56.148 INFO:teuthology.orchestra.run.smithi197.stdout: libcephfs2 x86_64 2:17.0.0-7401.g9366f375.el8 ceph 784 k 2021-09-02T07:24:56.149 INFO:teuthology.orchestra.run.smithi197.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-09-02T07:24:56.149 INFO:teuthology.orchestra.run.smithi197.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2021-09-02T07:24:56.149 INFO:teuthology.orchestra.run.smithi197.stdout: libradosstriper1 x86_64 2:17.0.0-7401.g9366f375.el8 ceph 482 k 2021-09-02T07:24:56.149 INFO:teuthology.orchestra.run.smithi197.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2021-09-02T07:24:56.150 INFO:teuthology.orchestra.run.smithi197.stdout: librgw2 x86_64 2:17.0.0-7401.g9366f375.el8 ceph 3.8 M 2021-09-02T07:24:56.150 INFO:teuthology.orchestra.run.smithi197.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-09-02T07:24:56.150 INFO:teuthology.orchestra.run.smithi197.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-09-02T07:24:56.150 INFO:teuthology.orchestra.run.smithi197.stdout: python3-ceph-argparse 2021-09-02T07:24:56.150 INFO:teuthology.orchestra.run.smithi197.stdout: x86_64 2:17.0.0-7401.g9366f375.el8 ceph 45 k 2021-09-02T07:24:56.151 INFO:teuthology.orchestra.run.smithi197.stdout: python3-ceph-common x86_64 2:17.0.0-7401.g9366f375.el8 ceph 78 k 2021-09-02T07:24:56.152 INFO:teuthology.orchestra.run.smithi197.stdout: python3-cephfs x86_64 2:17.0.0-7401.g9366f375.el8 ceph 178 k 2021-09-02T07:24:56.152 INFO:teuthology.orchestra.run.smithi197.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-09-02T07:24:56.152 INFO:teuthology.orchestra.run.smithi197.stdout: python3-rados x86_64 2:17.0.0-7401.g9366f375.el8 ceph 365 k 2021-09-02T07:24:56.153 INFO:teuthology.orchestra.run.smithi197.stdout: python3-rbd x86_64 2:17.0.0-7401.g9366f375.el8 ceph 346 k 2021-09-02T07:24:56.153 INFO:teuthology.orchestra.run.smithi197.stdout: python3-rgw x86_64 2:17.0.0-7401.g9366f375.el8 ceph 110 k 2021-09-02T07:24:56.153 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:24:56.154 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction Summary 2021-09-02T07:24:56.154 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:24:56.154 INFO:teuthology.orchestra.run.smithi197.stdout:Install 21 Packages 2021-09-02T07:24:56.155 INFO:teuthology.orchestra.run.smithi197.stdout:Upgrade 2 Packages 2021-09-02T07:24:56.155 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:24:56.156 INFO:teuthology.orchestra.run.smithi197.stdout:Total download size: 49 M 2021-09-02T07:24:56.156 INFO:teuthology.orchestra.run.smithi197.stdout:Downloading Packages: 2021-09-02T07:24:56.632 INFO:teuthology.orchestra.run.smithi175.stdout:(1/23): ceph-base-17.0.0-7401.g9366f375.el8.x86 8.1 MB/s | 5.2 MB 00:00 2021-09-02T07:24:56.686 INFO:teuthology.orchestra.run.smithi175.stdout:(2/23): ceph-selinux-17.0.0-7401.g9366f375.el8. 454 kB/s | 24 kB 00:00 2021-09-02T07:24:56.772 INFO:teuthology.orchestra.run.smithi175.stdout:(3/23): libcephfs2-17.0.0-7401.g9366f375.el8.x8 9.3 MB/s | 784 kB 00:00 2021-09-02T07:24:56.843 INFO:teuthology.orchestra.run.smithi175.stdout:(4/23): libradosstriper1-17.0.0-7401.g9366f375. 6.5 MB/s | 482 kB 00:00 2021-09-02T07:24:57.041 INFO:teuthology.orchestra.run.smithi175.stdout:(5/23): ceph-radosgw-17.0.0-7401.g9366f375.el8. 8.6 MB/s | 9.0 MB 00:01 2021-09-02T07:24:57.104 INFO:teuthology.orchestra.run.smithi175.stdout:(6/23): python3-ceph-argparse-17.0.0-7401.g9366 735 kB/s | 45 kB 00:00 2021-09-02T07:24:57.170 INFO:teuthology.orchestra.run.smithi175.stdout:(7/23): python3-ceph-common-17.0.0-7401.g9366f3 1.2 MB/s | 78 kB 00:00 2021-09-02T07:24:57.179 INFO:teuthology.orchestra.run.smithi175.stdout:(8/23): librgw2-17.0.0-7401.g9366f375.el8.x86_6 11 MB/s | 3.8 MB 00:00 2021-09-02T07:24:57.195 INFO:teuthology.orchestra.run.smithi175.stdout:(9/23): python3-cephfs-17.0.0-7401.g9366f375.el 6.0 MB/s | 178 kB 00:00 2021-09-02T07:24:57.218 INFO:teuthology.orchestra.run.smithi175.stdout:(10/23): python3-rados-17.0.0-7401.g9366f375.el 8.7 MB/s | 365 kB 00:00 2021-09-02T07:24:57.247 INFO:teuthology.orchestra.run.smithi175.stdout:(11/23): python3-rbd-17.0.0-7401.g9366f375.el8. 6.5 MB/s | 346 kB 00:00 2021-09-02T07:24:57.260 INFO:teuthology.orchestra.run.smithi175.stdout:(12/23): python3-rgw-17.0.0-7401.g9366f375.el8. 2.6 MB/s | 110 kB 00:00 2021-09-02T07:24:57.464 INFO:teuthology.orchestra.run.smithi175.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2021-09-02T07:24:57.466 INFO:teuthology.orchestra.run.smithi175.stdout:(14/23): librdkafka-0.11.4-1.el8.x86_64.rpm 1.6 MB/s | 353 kB 00:00 2021-09-02T07:24:57.466 INFO:teuthology.orchestra.run.smithi175.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 21 MB/s | 44 kB 00:00 2021-09-02T07:24:57.467 INFO:teuthology.orchestra.run.smithi175.stdout:(16/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 32 MB/s | 47 kB 00:00 2021-09-02T07:24:57.604 INFO:teuthology.orchestra.run.smithi175.stdout:(17/23): leveldb-1.22-1.el8.x86_64.rpm 1.3 MB/s | 181 kB 00:00 2021-09-02T07:24:57.617 INFO:teuthology.orchestra.run.smithi175.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 2.0 MB/s | 306 kB 00:00 2021-09-02T07:24:57.632 INFO:teuthology.orchestra.run.smithi175.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 2.1 MB/s | 59 kB 00:00 2021-09-02T07:24:57.643 INFO:teuthology.orchestra.run.smithi175.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 2.8 MB/s | 75 kB 00:00 2021-09-02T07:24:58.048 INFO:teuthology.orchestra.run.smithi175.stdout:(21/23): librados2-17.0.0-7401.g9366f375.el8.x8 8.2 MB/s | 3.4 MB 00:00 2021-09-02T07:24:58.058 INFO:teuthology.orchestra.run.smithi197.stdout:(1/23): ceph-base-17.0.0-7401.g9366f375.el8.x86 2.7 MB/s | 5.2 MB 00:01 2021-09-02T07:24:58.103 INFO:teuthology.orchestra.run.smithi197.stdout:(2/23): ceph-selinux-17.0.0-7401.g9366f375.el8. 547 kB/s | 24 kB 00:00 2021-09-02T07:24:58.232 INFO:teuthology.orchestra.run.smithi175.stdout:(22/23): librbd1-17.0.0-7401.g9366f375.el8.x86_ 6.5 MB/s | 3.9 MB 00:00 2021-09-02T07:24:58.277 INFO:teuthology.orchestra.run.smithi197.stdout:(3/23): libcephfs2-17.0.0-7401.g9366f375.el8.x8 4.4 MB/s | 784 kB 00:00 2021-09-02T07:24:58.374 INFO:teuthology.orchestra.run.smithi197.stdout:(4/23): libradosstriper1-17.0.0-7401.g9366f375. 4.9 MB/s | 482 kB 00:00 2021-09-02T07:24:58.470 INFO:teuthology.orchestra.run.smithi175.stdout:(23/23): ceph-common-17.0.0-7401.g9366f375.el8. 8.1 MB/s | 20 MB 00:02 2021-09-02T07:24:58.474 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:24:58.474 INFO:teuthology.orchestra.run.smithi175.stdout:Total 20 MB/s | 49 MB 00:02 2021-09-02T07:24:58.478 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2021-09-02T07:24:58.522 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2021-09-02T07:24:58.523 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2021-09-02T07:24:58.916 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2021-09-02T07:24:58.919 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2021-09-02T07:24:58.964 INFO:teuthology.orchestra.run.smithi197.stdout:(5/23): librgw2-17.0.0-7401.g9366f375.el8.x86_6 6.5 MB/s | 3.8 MB 00:00 2021-09-02T07:24:59.004 INFO:teuthology.orchestra.run.smithi197.stdout:(6/23): python3-ceph-argparse-17.0.0-7401.g9366 1.1 MB/s | 45 kB 00:00 2021-09-02T07:24:59.049 INFO:teuthology.orchestra.run.smithi197.stdout:(7/23): python3-ceph-common-17.0.0-7401.g9366f3 1.7 MB/s | 78 kB 00:00 2021-09-02T07:24:59.168 INFO:teuthology.orchestra.run.smithi197.stdout:(8/23): ceph-common-17.0.0-7401.g9366f375.el8.x 6.6 MB/s | 20 MB 00:03 2021-09-02T07:24:59.219 INFO:teuthology.orchestra.run.smithi197.stdout:(9/23): python3-cephfs-17.0.0-7401.g9366f375.el 1.0 MB/s | 178 kB 00:00 2021-09-02T07:24:59.221 INFO:teuthology.orchestra.run.smithi197.stdout:(10/23): python3-rados-17.0.0-7401.g9366f375.el 6.8 MB/s | 365 kB 00:00 2021-09-02T07:24:59.246 INFO:teuthology.orchestra.run.smithi197.stdout:(11/23): python3-rgw-17.0.0-7401.g9366f375.el8. 4.3 MB/s | 110 kB 00:00 2021-09-02T07:24:59.268 INFO:teuthology.orchestra.run.smithi197.stdout:(12/23): python3-rbd-17.0.0-7401.g9366f375.el8. 6.8 MB/s | 346 kB 00:00 2021-09-02T07:24:59.281 INFO:teuthology.orchestra.run.smithi197.stdout:(13/23): librdkafka-0.11.4-1.el8.x86_64.rpm 9.9 MB/s | 353 kB 00:00 2021-09-02T07:24:59.282 INFO:teuthology.orchestra.run.smithi197.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 30 MB/s | 44 kB 00:00 2021-09-02T07:24:59.284 INFO:teuthology.orchestra.run.smithi197.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 35 MB/s | 47 kB 00:00 2021-09-02T07:24:59.300 INFO:teuthology.orchestra.run.smithi197.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 7.9 MB/s | 259 kB 00:00 2021-09-02T07:24:59.428 INFO:teuthology.orchestra.run.smithi197.stdout:(17/23): gperftools-libs-2.7-9.el8.x86_64.rpm 2.1 MB/s | 306 kB 00:00 2021-09-02T07:24:59.435 INFO:teuthology.orchestra.run.smithi197.stdout:(18/23): leveldb-1.22-1.el8.x86_64.rpm 1.3 MB/s | 181 kB 00:00 2021-09-02T07:24:59.453 INFO:teuthology.orchestra.run.smithi197.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 2.4 MB/s | 59 kB 00:00 2021-09-02T07:24:59.463 INFO:teuthology.orchestra.run.smithi197.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 2.6 MB/s | 75 kB 00:00 2021-09-02T07:24:59.562 INFO:teuthology.orchestra.run.smithi197.stdout:(21/23): ceph-radosgw-17.0.0-7401.g9366f375.el8 2.6 MB/s | 9.0 MB 00:03 2021-09-02T07:24:59.659 INFO:teuthology.orchestra.run.smithi197.stdout:(22/23): librbd1-17.0.0-7401.g9366f375.el8.x86_ 20 MB/s | 3.9 MB 00:00 2021-09-02T07:24:59.859 INFO:teuthology.orchestra.run.smithi197.stdout:(23/23): librados2-17.0.0-7401.g9366f375.el8.x8 8.4 MB/s | 3.4 MB 00:00 2021-09-02T07:24:59.864 INFO:teuthology.orchestra.run.smithi197.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:24:59.864 INFO:teuthology.orchestra.run.smithi197.stdout:Total 13 MB/s | 49 MB 00:03 2021-09-02T07:24:59.868 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction check 2021-09-02T07:24:59.869 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2021-09-02T07:24:59.912 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction check succeeded. 2021-09-02T07:24:59.912 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction test 2021-09-02T07:25:00.053 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2021-09-02T07:25:00.197 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2021-09-02T07:25:00.341 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction test succeeded. 2021-09-02T07:25:00.348 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction 2021-09-02T07:25:00.596 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2021-09-02T07:25:00.767 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : librados2-2:17.0.0-7401.g9366f375.el8.x86_64 3/25 2021-09-02T07:25:00.843 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librados2-2:17.0.0-7401.g9366f375.el8.x86_64 3/25 2021-09-02T07:25:01.055 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rados-2:17.0.0-7401.g9366f375.el8.x86_64 4/25 2021-09-02T07:25:01.181 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephfs2-2:17.0.0-7401.g9366f375.el8.x86_64 5/25 2021-09-02T07:25:01.314 INFO:teuthology.orchestra.run.smithi197.stdout: Preparing : 1/1 2021-09-02T07:25:01.469 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2021-09-02T07:25:01.592 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libcephfs2-2:17.0.0-7401.g9366f375.el8.x86_64 5/25 2021-09-02T07:25:01.607 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2021-09-02T07:25:01.780 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : librbd1-2:17.0.0-7401.g9366f375.el8.x86_64 6/25 2021-09-02T07:25:01.849 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librbd1-2:17.0.0-7401.g9366f375.el8.x86_64 6/25 2021-09-02T07:25:01.996 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2021-09-02T07:25:02.011 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2021-09-02T07:25:02.147 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2021-09-02T07:25:02.157 INFO:teuthology.orchestra.run.smithi197.stdout: Upgrading : librados2-2:17.0.0-7401.g9366f375.el8.x86_64 3/25 2021-09-02T07:25:02.255 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: librados2-2:17.0.0-7401.g9366f375.el8.x86_64 3/25 2021-09-02T07:25:02.441 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-rados-2:17.0.0-7401.g9366f375.el8.x86_64 4/25 2021-09-02T07:25:02.538 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : libcephfs2-2:17.0.0-7401.g9366f375.el8.x86_64 5/25 2021-09-02T07:25:02.592 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2021-09-02T07:25:02.755 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librgw2-2:17.0.0-7401.g9366f375.el8.x86_64 9/25 2021-09-02T07:25:02.821 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librgw2-2:17.0.0-7401.g9366f375.el8.x86_64 9/25 2021-09-02T07:25:02.963 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2021-09-02T07:25:02.965 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: libcephfs2-2:17.0.0-7401.g9366f375.el8.x86_64 5/25 2021-09-02T07:25:03.090 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-ceph-argparse-2:17.0.0-7401.g9366f375.el8. 11/25 2021-09-02T07:25:03.140 INFO:teuthology.orchestra.run.smithi197.stdout: Upgrading : librbd1-2:17.0.0-7401.g9366f375.el8.x86_64 6/25 2021-09-02T07:25:03.207 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: librbd1-2:17.0.0-7401.g9366f375.el8.x86_64 6/25 2021-09-02T07:25:03.250 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cephfs-2:17.0.0-7401.g9366f375.el8.x86_64 12/25 2021-09-02T07:25:03.331 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2021-09-02T07:25:03.484 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2021-09-02T07:25:03.485 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rgw-2:17.0.0-7401.g9366f375.el8.x86_64 13/25 2021-09-02T07:25:03.665 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rbd-2:17.0.0-7401.g9366f375.el8.x86_64 14/25 2021-09-02T07:25:03.805 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libradosstriper1-2:17.0.0-7401.g9366f375.el8.x86_6 15/25 2021-09-02T07:25:03.876 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libradosstriper1-2:17.0.0-7401.g9366f375.el8.x86_6 15/25 2021-09-02T07:25:03.892 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2021-09-02T07:25:04.047 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2021-09-02T07:25:04.070 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : librgw2-2:17.0.0-7401.g9366f375.el8.x86_64 9/25 2021-09-02T07:25:04.135 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: librgw2-2:17.0.0-7401.g9366f375.el8.x86_64 9/25 2021-09-02T07:25:04.222 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2021-09-02T07:25:04.247 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2021-09-02T07:25:04.411 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2021-09-02T07:25:04.434 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-ceph-argparse-2:17.0.0-7401.g9366f375.el8. 11/25 2021-09-02T07:25:04.572 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-ceph-common-2:17.0.0-7401.g9366f375.el8.x8 19/25 2021-09-02T07:25:04.593 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-cephfs-2:17.0.0-7401.g9366f375.el8.x86_64 12/25 2021-09-02T07:25:04.791 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-rgw-2:17.0.0-7401.g9366f375.el8.x86_64 13/25 2021-09-02T07:25:04.950 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-rbd-2:17.0.0-7401.g9366f375.el8.x86_64 14/25 2021-09-02T07:25:05.081 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : libradosstriper1-2:17.0.0-7401.g9366f375.el8.x86_6 15/25 2021-09-02T07:25:05.155 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: libradosstriper1-2:17.0.0-7401.g9366f375.el8.x86_6 15/25 2021-09-02T07:25:05.298 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2021-09-02T07:25:05.457 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2021-09-02T07:25:05.617 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2021-09-02T07:25:05.761 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-ceph-common-2:17.0.0-7401.g9366f375.el8.x8 19/25 2021-09-02T07:25:08.598 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-common-2:17.0.0-7401.g9366f375.el8.x86_64 20/25 2021-09-02T07:25:08.771 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-common-2:17.0.0-7401.g9366f375.el8.x86_64 20/25 2021-09-02T07:25:09.356 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-common-2:17.0.0-7401.g9366f375.el8.x86_64 20/25 2021-09-02T07:25:09.564 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-base-2:17.0.0-7401.g9366f375.el8.x86_64 21/25 2021-09-02T07:25:09.653 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-base-2:17.0.0-7401.g9366f375.el8.x86_64 21/25 2021-09-02T07:25:09.787 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-selinux-2:17.0.0-7401.g9366f375.el8.x86_64 22/25 2021-09-02T07:25:10.037 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: ceph-common-2:17.0.0-7401.g9366f375.el8.x86_64 20/25 2021-09-02T07:25:10.206 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : ceph-common-2:17.0.0-7401.g9366f375.el8.x86_64 20/25 2021-09-02T07:25:10.814 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: ceph-common-2:17.0.0-7401.g9366f375.el8.x86_64 20/25 2021-09-02T07:25:11.051 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : ceph-base-2:17.0.0-7401.g9366f375.el8.x86_64 21/25 2021-09-02T07:25:11.161 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: ceph-base-2:17.0.0-7401.g9366f375.el8.x86_64 21/25 2021-09-02T07:25:11.264 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : ceph-selinux-2:17.0.0-7401.g9366f375.el8.x86_64 22/25 2021-09-02T07:25:25.298 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-selinux-2:17.0.0-7401.g9366f375.el8.x86_64 22/25 2021-09-02T07:25:25.299 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /sys 2021-09-02T07:25:25.299 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /proc 2021-09-02T07:25:25.299 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /dev 2021-09-02T07:25:25.299 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /run 2021-09-02T07:25:25.300 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /mnt 2021-09-02T07:25:25.300 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /var/tmp 2021-09-02T07:25:25.301 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /home 2021-09-02T07:25:25.301 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /tmp 2021-09-02T07:25:25.301 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /dev 2021-09-02T07:25:25.301 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:25:26.412 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-radosgw-2:17.0.0-7401.g9366f375.el8.x86_64 23/25 2021-09-02T07:25:26.482 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-7401.g9366f375.el8.x86_64 23/25 2021-09-02T07:25:26.483 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-09-02T07:25:26.686 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-09-02T07:25:26.687 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2021-09-02T07:25:27.219 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: ceph-selinux-2:17.0.0-7401.g9366f375.el8.x86_64 22/25 2021-09-02T07:25:27.219 INFO:teuthology.orchestra.run.smithi197.stdout:skipping the directory /sys 2021-09-02T07:25:27.219 INFO:teuthology.orchestra.run.smithi197.stdout:skipping the directory /proc 2021-09-02T07:25:27.220 INFO:teuthology.orchestra.run.smithi197.stdout:skipping the directory /dev 2021-09-02T07:25:27.220 INFO:teuthology.orchestra.run.smithi197.stdout:skipping the directory /run 2021-09-02T07:25:27.220 INFO:teuthology.orchestra.run.smithi197.stdout:skipping the directory /mnt 2021-09-02T07:25:27.221 INFO:teuthology.orchestra.run.smithi197.stdout:skipping the directory /var/tmp 2021-09-02T07:25:27.221 INFO:teuthology.orchestra.run.smithi197.stdout:skipping the directory /home 2021-09-02T07:25:27.221 INFO:teuthology.orchestra.run.smithi197.stdout:skipping the directory /tmp 2021-09-02T07:25:27.222 INFO:teuthology.orchestra.run.smithi197.stdout:skipping the directory /dev 2021-09-02T07:25:27.222 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:25:27.396 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2021-09-02T07:25:27.397 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-base-2:17.0.0-7401.g9366f375.el8.x86_64 1/25 2021-09-02T07:25:27.397 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-common-2:17.0.0-7401.g9366f375.el8.x86_64 2/25 2021-09-02T07:25:27.397 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-radosgw-2:17.0.0-7401.g9366f375.el8.x86_64 3/25 2021-09-02T07:25:27.397 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-selinux-2:17.0.0-7401.g9366f375.el8.x86_64 4/25 2021-09-02T07:25:27.397 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephfs2-2:17.0.0-7401.g9366f375.el8.x86_64 5/25 2021-09-02T07:25:27.398 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libradosstriper1-2:17.0.0-7401.g9366f375.el8.x86_6 6/25 2021-09-02T07:25:27.398 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librgw2-2:17.0.0-7401.g9366f375.el8.x86_64 7/25 2021-09-02T07:25:27.398 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-ceph-argparse-2:17.0.0-7401.g9366f375.el8. 8/25 2021-09-02T07:25:27.398 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-ceph-common-2:17.0.0-7401.g9366f375.el8.x8 9/25 2021-09-02T07:25:27.399 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cephfs-2:17.0.0-7401.g9366f375.el8.x86_64 10/25 2021-09-02T07:25:27.399 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rados-2:17.0.0-7401.g9366f375.el8.x86_64 11/25 2021-09-02T07:25:27.399 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rbd-2:17.0.0-7401.g9366f375.el8.x86_64 12/25 2021-09-02T07:25:27.399 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rgw-2:17.0.0-7401.g9366f375.el8.x86_64 13/25 2021-09-02T07:25:27.399 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/25 2021-09-02T07:25:27.400 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2021-09-02T07:25:27.400 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2021-09-02T07:25:27.400 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2021-09-02T07:25:27.400 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2021-09-02T07:25:27.401 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2021-09-02T07:25:27.401 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2021-09-02T07:25:27.401 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2021-09-02T07:25:27.401 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados2-2:17.0.0-7401.g9366f375.el8.x86_64 22/25 2021-09-02T07:25:27.401 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2021-09-02T07:25:27.402 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librbd1-2:17.0.0-7401.g9366f375.el8.x86_64 24/25 2021-09-02T07:25:28.151 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2021-09-02T07:25:28.151 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:25:28.151 INFO:teuthology.orchestra.run.smithi175.stdout:Upgraded: 2021-09-02T07:25:28.152 INFO:teuthology.orchestra.run.smithi175.stdout: librados2-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:28.152 INFO:teuthology.orchestra.run.smithi175.stdout: librbd1-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:28.152 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:25:28.152 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2021-09-02T07:25:28.153 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-base-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:28.153 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-common-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:28.153 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-radosgw-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:28.154 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-selinux-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:28.154 INFO:teuthology.orchestra.run.smithi175.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-09-02T07:25:28.154 INFO:teuthology.orchestra.run.smithi175.stdout: leveldb-1.22-1.el8.x86_64 2021-09-02T07:25:28.155 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs2-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:28.155 INFO:teuthology.orchestra.run.smithi175.stdout: liboath-2.6.2-3.el8.x86_64 2021-09-02T07:25:28.155 INFO:teuthology.orchestra.run.smithi175.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-09-02T07:25:28.155 INFO:teuthology.orchestra.run.smithi175.stdout: libradosstriper1-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:28.156 INFO:teuthology.orchestra.run.smithi175.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-09-02T07:25:28.157 INFO:teuthology.orchestra.run.smithi175.stdout: librgw2-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:28.157 INFO:teuthology.orchestra.run.smithi175.stdout: libunwind-1.3.1-3.el8.x86_64 2021-09-02T07:25:28.157 INFO:teuthology.orchestra.run.smithi175.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-09-02T07:25:28.157 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-argparse-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:28.158 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-common-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:28.158 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cephfs-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:28.158 INFO:teuthology.orchestra.run.smithi175.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-09-02T07:25:28.158 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rados-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:28.159 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rbd-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:28.159 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rgw-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:28.159 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:25:28.160 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:25:28.326 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-test 2021-09-02T07:25:28.331 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : ceph-radosgw-2:17.0.0-7401.g9366f375.el8.x86_64 23/25 2021-09-02T07:25:28.406 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-7401.g9366f375.el8.x86_64 23/25 2021-09-02T07:25:28.406 INFO:teuthology.orchestra.run.smithi197.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-09-02T07:25:28.605 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-09-02T07:25:28.605 INFO:teuthology.orchestra.run.smithi197.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2021-09-02T07:25:28.782 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:35 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:25:29.303 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2021-09-02T07:25:29.304 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : ceph-base-2:17.0.0-7401.g9366f375.el8.x86_64 1/25 2021-09-02T07:25:29.304 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : ceph-common-2:17.0.0-7401.g9366f375.el8.x86_64 2/25 2021-09-02T07:25:29.305 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : ceph-radosgw-2:17.0.0-7401.g9366f375.el8.x86_64 3/25 2021-09-02T07:25:29.305 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : ceph-selinux-2:17.0.0-7401.g9366f375.el8.x86_64 4/25 2021-09-02T07:25:29.306 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : libcephfs2-2:17.0.0-7401.g9366f375.el8.x86_64 5/25 2021-09-02T07:25:29.306 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : libradosstriper1-2:17.0.0-7401.g9366f375.el8.x86_6 6/25 2021-09-02T07:25:29.306 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : librgw2-2:17.0.0-7401.g9366f375.el8.x86_64 7/25 2021-09-02T07:25:29.309 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-ceph-argparse-2:17.0.0-7401.g9366f375.el8. 8/25 2021-09-02T07:25:29.309 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-ceph-common-2:17.0.0-7401.g9366f375.el8.x8 9/25 2021-09-02T07:25:29.310 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-cephfs-2:17.0.0-7401.g9366f375.el8.x86_64 10/25 2021-09-02T07:25:29.310 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-rados-2:17.0.0-7401.g9366f375.el8.x86_64 11/25 2021-09-02T07:25:29.311 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-rbd-2:17.0.0-7401.g9366f375.el8.x86_64 12/25 2021-09-02T07:25:29.311 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-rgw-2:17.0.0-7401.g9366f375.el8.x86_64 13/25 2021-09-02T07:25:29.311 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/25 2021-09-02T07:25:29.311 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2021-09-02T07:25:29.312 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2021-09-02T07:25:29.312 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2021-09-02T07:25:29.312 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2021-09-02T07:25:29.312 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2021-09-02T07:25:29.313 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2021-09-02T07:25:29.313 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2021-09-02T07:25:29.313 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : librados2-2:17.0.0-7401.g9366f375.el8.x86_64 22/25 2021-09-02T07:25:29.314 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2021-09-02T07:25:29.314 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : librbd1-2:17.0.0-7401.g9366f375.el8.x86_64 24/25 2021-09-02T07:25:29.740 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:25:29.741 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:25:29.741 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2021-09-02T07:25:29.742 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:25:29.742 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2021-09-02T07:25:29.742 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-test x86_64 2:17.0.0-7401.g9366f375.el8 ceph 44 M 2021-09-02T07:25:29.742 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2021-09-02T07:25:29.743 INFO:teuthology.orchestra.run.smithi175.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2021-09-02T07:25:29.743 INFO:teuthology.orchestra.run.smithi175.stdout: libcephsqlite x86_64 2:17.0.0-7401.g9366f375.el8 ceph 170 k 2021-09-02T07:25:29.743 INFO:teuthology.orchestra.run.smithi175.stdout: libxslt x86_64 1.1.32-6.el8 CentOS-Base 250 k 2021-09-02T07:25:29.743 INFO:teuthology.orchestra.run.smithi175.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-09-02T07:25:29.744 INFO:teuthology.orchestra.run.smithi175.stdout: socat x86_64 1.7.3.3-2.el8 CentOS-AppStream 302 k 2021-09-02T07:25:29.744 INFO:teuthology.orchestra.run.smithi175.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-09-02T07:25:29.744 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:25:29.744 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2021-09-02T07:25:29.744 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:25:29.745 INFO:teuthology.orchestra.run.smithi175.stdout:Install 7 Packages 2021-09-02T07:25:29.745 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:25:29.746 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 45 M 2021-09-02T07:25:29.746 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 212 M 2021-09-02T07:25:29.746 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2021-09-02T07:25:29.781 INFO:teuthology.orchestra.run.smithi175.stdout:(1/7): jq-1.5-12.el8.x86_64.rpm 4.2 MB/s | 161 kB 00:00 2021-09-02T07:25:29.786 INFO:teuthology.orchestra.run.smithi175.stdout:(2/7): oniguruma-6.8.2-2.el8.x86_64.rpm 51 MB/s | 187 kB 00:00 2021-09-02T07:25:29.789 INFO:teuthology.orchestra.run.smithi175.stdout:(3/7): socat-1.7.3.3-2.el8.x86_64.rpm 69 MB/s | 302 kB 00:00 2021-09-02T07:25:29.793 INFO:teuthology.orchestra.run.smithi175.stdout:(4/7): libxslt-1.1.32-6.el8.x86_64.rpm 70 MB/s | 250 kB 00:00 2021-09-02T07:25:29.896 INFO:teuthology.orchestra.run.smithi175.stdout:(5/7): libcephsqlite-17.0.0-7401.g9366f375.el8. 1.1 MB/s | 170 kB 00:00 2021-09-02T07:25:29.904 INFO:teuthology.orchestra.run.smithi175.stdout:(6/7): xmlstarlet-1.6.1-11.el8.x86_64.rpm 622 kB/s | 69 kB 00:00 2021-09-02T07:25:30.012 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2021-09-02T07:25:30.012 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:25:30.013 INFO:teuthology.orchestra.run.smithi197.stdout:Upgraded: 2021-09-02T07:25:30.013 INFO:teuthology.orchestra.run.smithi197.stdout: librados2-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:30.013 INFO:teuthology.orchestra.run.smithi197.stdout: librbd1-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:30.013 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:25:30.014 INFO:teuthology.orchestra.run.smithi197.stdout:Installed: 2021-09-02T07:25:30.014 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-base-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:30.014 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-common-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:30.014 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-radosgw-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:30.015 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-selinux-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:30.015 INFO:teuthology.orchestra.run.smithi197.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-09-02T07:25:30.015 INFO:teuthology.orchestra.run.smithi197.stdout: leveldb-1.22-1.el8.x86_64 2021-09-02T07:25:30.015 INFO:teuthology.orchestra.run.smithi197.stdout: libcephfs2-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:30.016 INFO:teuthology.orchestra.run.smithi197.stdout: liboath-2.6.2-3.el8.x86_64 2021-09-02T07:25:30.016 INFO:teuthology.orchestra.run.smithi197.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-09-02T07:25:30.016 INFO:teuthology.orchestra.run.smithi197.stdout: libradosstriper1-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:30.016 INFO:teuthology.orchestra.run.smithi197.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-09-02T07:25:30.017 INFO:teuthology.orchestra.run.smithi197.stdout: librgw2-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:30.017 INFO:teuthology.orchestra.run.smithi197.stdout: libunwind-1.3.1-3.el8.x86_64 2021-09-02T07:25:30.017 INFO:teuthology.orchestra.run.smithi197.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-09-02T07:25:30.017 INFO:teuthology.orchestra.run.smithi197.stdout: python3-ceph-argparse-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:30.018 INFO:teuthology.orchestra.run.smithi197.stdout: python3-ceph-common-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:30.018 INFO:teuthology.orchestra.run.smithi197.stdout: python3-cephfs-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:30.018 INFO:teuthology.orchestra.run.smithi197.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-09-02T07:25:30.018 INFO:teuthology.orchestra.run.smithi197.stdout: python3-rados-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:30.019 INFO:teuthology.orchestra.run.smithi197.stdout: python3-rbd-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:30.019 INFO:teuthology.orchestra.run.smithi197.stdout: python3-rgw-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:30.019 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:25:30.019 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:25:30.243 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install ceph-test 2021-09-02T07:25:30.699 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:00:37 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:25:31.651 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:25:31.652 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:25:31.652 INFO:teuthology.orchestra.run.smithi197.stdout: Package Arch Version Repository Size 2021-09-02T07:25:31.653 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:25:31.653 INFO:teuthology.orchestra.run.smithi197.stdout:Installing: 2021-09-02T07:25:31.654 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-test x86_64 2:17.0.0-7401.g9366f375.el8 ceph 44 M 2021-09-02T07:25:31.654 INFO:teuthology.orchestra.run.smithi197.stdout:Installing dependencies: 2021-09-02T07:25:31.655 INFO:teuthology.orchestra.run.smithi197.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2021-09-02T07:25:31.655 INFO:teuthology.orchestra.run.smithi197.stdout: libcephsqlite x86_64 2:17.0.0-7401.g9366f375.el8 ceph 170 k 2021-09-02T07:25:31.656 INFO:teuthology.orchestra.run.smithi197.stdout: libxslt x86_64 1.1.32-6.el8 CentOS-Base 250 k 2021-09-02T07:25:31.656 INFO:teuthology.orchestra.run.smithi197.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-09-02T07:25:31.657 INFO:teuthology.orchestra.run.smithi197.stdout: socat x86_64 1.7.3.3-2.el8 CentOS-AppStream 302 k 2021-09-02T07:25:31.657 INFO:teuthology.orchestra.run.smithi197.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-09-02T07:25:31.658 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:25:31.658 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction Summary 2021-09-02T07:25:31.659 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:25:31.659 INFO:teuthology.orchestra.run.smithi197.stdout:Install 7 Packages 2021-09-02T07:25:31.659 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:25:31.661 INFO:teuthology.orchestra.run.smithi197.stdout:Total download size: 45 M 2021-09-02T07:25:31.661 INFO:teuthology.orchestra.run.smithi197.stdout:Installed size: 212 M 2021-09-02T07:25:31.661 INFO:teuthology.orchestra.run.smithi197.stdout:Downloading Packages: 2021-09-02T07:25:31.696 INFO:teuthology.orchestra.run.smithi197.stdout:(1/7): jq-1.5-12.el8.x86_64.rpm 3.8 MB/s | 161 kB 00:00 2021-09-02T07:25:31.699 INFO:teuthology.orchestra.run.smithi197.stdout:(2/7): oniguruma-6.8.2-2.el8.x86_64.rpm 65 MB/s | 187 kB 00:00 2021-09-02T07:25:31.703 INFO:teuthology.orchestra.run.smithi197.stdout:(3/7): socat-1.7.3.3-2.el8.x86_64.rpm 71 MB/s | 302 kB 00:00 2021-09-02T07:25:31.708 INFO:teuthology.orchestra.run.smithi197.stdout:(4/7): libxslt-1.1.32-6.el8.x86_64.rpm 65 MB/s | 250 kB 00:00 2021-09-02T07:25:31.823 INFO:teuthology.orchestra.run.smithi197.stdout:(5/7): xmlstarlet-1.6.1-11.el8.x86_64.rpm 600 kB/s | 69 kB 00:00 2021-09-02T07:25:31.845 INFO:teuthology.orchestra.run.smithi197.stdout:(6/7): libcephsqlite-17.0.0-7401.g9366f375.el8. 903 kB/s | 170 kB 00:00 2021-09-02T07:25:31.901 INFO:teuthology.orchestra.run.smithi175.stdout:(7/7): ceph-test-17.0.0-7401.g9366f375.el8.x86_ 21 MB/s | 44 MB 00:02 2021-09-02T07:25:31.904 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:25:31.904 INFO:teuthology.orchestra.run.smithi175.stdout:Total 21 MB/s | 45 MB 00:02 2021-09-02T07:25:31.908 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2021-09-02T07:25:31.931 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2021-09-02T07:25:31.931 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2021-09-02T07:25:32.178 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2021-09-02T07:25:32.181 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2021-09-02T07:25:32.659 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2021-09-02T07:25:32.830 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libxslt-1.1.32-6.el8.x86_64 1/7 2021-09-02T07:25:32.988 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/7 2021-09-02T07:25:33.153 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 3/7 2021-09-02T07:25:33.299 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-09-02T07:25:33.363 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 4/7 2021-09-02T07:25:33.534 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : jq-1.5-12.el8.x86_64 5/7 2021-09-02T07:25:33.683 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephsqlite-2:17.0.0-7401.g9366f375.el8.x86_64 6/7 2021-09-02T07:25:34.376 INFO:teuthology.orchestra.run.smithi197.stdout:(7/7): ceph-test-17.0.0-7401.g9366f375.el8.x86_ 16 MB/s | 44 MB 00:02 2021-09-02T07:25:34.380 INFO:teuthology.orchestra.run.smithi197.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:25:34.381 INFO:teuthology.orchestra.run.smithi197.stdout:Total 17 MB/s | 45 MB 00:02 2021-09-02T07:25:34.385 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction check 2021-09-02T07:25:34.408 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction check succeeded. 2021-09-02T07:25:34.408 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction test 2021-09-02T07:25:34.657 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction test succeeded. 2021-09-02T07:25:34.660 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction 2021-09-02T07:25:35.320 INFO:teuthology.orchestra.run.smithi197.stdout: Preparing : 1/1 2021-09-02T07:25:35.622 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : libxslt-1.1.32-6.el8.x86_64 1/7 2021-09-02T07:25:35.772 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/7 2021-09-02T07:25:35.896 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 3/7 2021-09-02T07:25:36.101 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-09-02T07:25:36.176 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 4/7 2021-09-02T07:25:36.319 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : jq-1.5-12.el8.x86_64 5/7 2021-09-02T07:25:36.444 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : libcephsqlite-2:17.0.0-7401.g9366f375.el8.x86_64 6/7 2021-09-02T07:25:38.059 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libcephsqlite-2:17.0.0-7401.g9366f375.el8.x86_64 6/7 2021-09-02T07:25:38.289 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-test-2:17.0.0-7401.g9366f375.el8.x86_64 7/7 2021-09-02T07:25:38.509 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-test-2:17.0.0-7401.g9366f375.el8.x86_64 7/7 2021-09-02T07:25:38.510 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-test-2:17.0.0-7401.g9366f375.el8.x86_64 1/7 2021-09-02T07:25:38.510 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephsqlite-2:17.0.0-7401.g9366f375.el8.x86_64 2/7 2021-09-02T07:25:38.510 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : jq-1.5-12.el8.x86_64 3/7 2021-09-02T07:25:38.510 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-09-02T07:25:38.511 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 5/7 2021-09-02T07:25:38.511 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libxslt-1.1.32-6.el8.x86_64 6/7 2021-09-02T07:25:38.891 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 7/7 2021-09-02T07:25:38.891 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:25:38.891 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2021-09-02T07:25:38.892 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-test-2:17.0.0-7401.g9366f375.el8.x86_64 jq-1.5-12.el8.x86_64 2021-09-02T07:25:38.892 INFO:teuthology.orchestra.run.smithi175.stdout: libcephsqlite-2:17.0.0-7401.g9366f375.el8.x86_64 libxslt-1.1.32-6.el8.x86_64 2021-09-02T07:25:38.892 INFO:teuthology.orchestra.run.smithi175.stdout: oniguruma-6.8.2-2.el8.x86_64 socat-1.7.3.3-2.el8.x86_64 2021-09-02T07:25:38.892 INFO:teuthology.orchestra.run.smithi175.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-09-02T07:25:38.893 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:25:38.893 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:25:39.061 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph 2021-09-02T07:25:39.504 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:46 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:25:40.486 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:25:40.490 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================= 2021-09-02T07:25:40.491 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2021-09-02T07:25:40.491 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================= 2021-09-02T07:25:40.491 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2021-09-02T07:25:40.491 INFO:teuthology.orchestra.run.smithi175.stdout: ceph x86_64 2:17.0.0-7401.g9366f375.el8 ceph 6.4 k 2021-09-02T07:25:40.492 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2021-09-02T07:25:40.492 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mds x86_64 2:17.0.0-7401.g9366f375.el8 ceph 2.3 M 2021-09-02T07:25:40.492 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr x86_64 2:17.0.0-7401.g9366f375.el8 ceph 1.5 M 2021-09-02T07:25:40.492 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-modules-core noarch 2:17.0.0-7401.g9366f375.el8 ceph-noarch 214 k 2021-09-02T07:25:40.493 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mon x86_64 2:17.0.0-7401.g9366f375.el8 ceph 3.9 M 2021-09-02T07:25:40.493 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-osd x86_64 2:17.0.0-7401.g9366f375.el8 ceph 17 M 2021-09-02T07:25:40.493 INFO:teuthology.orchestra.run.smithi175.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-09-02T07:25:40.493 INFO:teuthology.orchestra.run.smithi175.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-09-02T07:25:40.493 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-09-02T07:25:40.494 INFO:teuthology.orchestra.run.smithi175.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-09-02T07:25:40.494 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-09-02T07:25:40.494 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-09-02T07:25:40.494 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cryptography x86_64 3.2.1-4.el8 CentOS-Base 559 k 2021-09-02T07:25:40.495 INFO:teuthology.orchestra.run.smithi175.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-09-02T07:25:40.495 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-09-02T07:25:40.495 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-09-02T07:25:40.495 INFO:teuthology.orchestra.run.smithi175.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-09-02T07:25:40.496 INFO:teuthology.orchestra.run.smithi175.stdout: python3-lxml x86_64 4.2.3-2.el8 CentOS-AppStream 1.5 M 2021-09-02T07:25:40.496 INFO:teuthology.orchestra.run.smithi175.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-09-02T07:25:40.496 INFO:teuthology.orchestra.run.smithi175.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-09-02T07:25:40.496 INFO:teuthology.orchestra.run.smithi175.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-09-02T07:25:40.496 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-09-02T07:25:40.497 INFO:teuthology.orchestra.run.smithi175.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-09-02T07:25:40.497 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2021-09-02T07:25:40.497 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-09-02T07:25:40.497 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-09-02T07:25:40.498 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-09-02T07:25:40.498 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-09-02T07:25:40.498 INFO:teuthology.orchestra.run.smithi175.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-09-02T07:25:40.498 INFO:teuthology.orchestra.run.smithi175.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-09-02T07:25:40.498 INFO:teuthology.orchestra.run.smithi175.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-09-02T07:25:40.499 INFO:teuthology.orchestra.run.smithi175.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-09-02T07:25:40.499 INFO:teuthology.orchestra.run.smithi175.stdout: python3-urllib3 noarch 1.24.2-5.el8 CentOS-Base 177 k 2021-09-02T07:25:40.499 INFO:teuthology.orchestra.run.smithi175.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-09-02T07:25:40.499 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webencodings noarch 0.5.1-6.el8 CentOS-AppStream 27 k 2021-09-02T07:25:40.500 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-09-02T07:25:40.500 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-09-02T07:25:40.500 INFO:teuthology.orchestra.run.smithi175.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-09-02T07:25:40.500 INFO:teuthology.orchestra.run.smithi175.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-09-02T07:25:40.500 INFO:teuthology.orchestra.run.smithi175.stdout:Installing weak dependencies: 2021-09-02T07:25:40.501 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-09-02T07:25:40.501 INFO:teuthology.orchestra.run.smithi175.stdout: python3-html5lib noarch 1:0.999999999-6.el8 CentOS-AppStream 214 k 2021-09-02T07:25:40.501 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:25:40.501 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2021-09-02T07:25:40.502 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================= 2021-09-02T07:25:40.502 INFO:teuthology.orchestra.run.smithi175.stdout:Install 41 Packages 2021-09-02T07:25:40.502 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:25:40.503 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 31 M 2021-09-02T07:25:40.503 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 121 M 2021-09-02T07:25:40.503 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2021-09-02T07:25:40.621 INFO:teuthology.orchestra.run.smithi175.stdout:(1/41): ceph-17.0.0-7401.g9366f375.el8.x86_64.r 52 kB/s | 6.4 kB 00:00 2021-09-02T07:25:41.061 INFO:teuthology.orchestra.run.smithi175.stdout:(2/41): ceph-mgr-17.0.0-7401.g9366f375.el8.x86_ 2.7 MB/s | 1.5 MB 00:00 2021-09-02T07:25:41.152 INFO:teuthology.orchestra.run.smithi175.stdout:(3/41): ceph-mds-17.0.0-7401.g9366f375.el8.x86_ 3.4 MB/s | 2.3 MB 00:00 2021-09-02T07:25:41.228 INFO:teuthology.orchestra.run.smithi175.stdout:(4/41): ceph-mgr-modules-core-17.0.0-7401.g9366 2.8 MB/s | 214 kB 00:00 2021-09-02T07:25:41.282 INFO:teuthology.orchestra.run.smithi175.stdout:(5/41): python3-html5lib-0.999999999-6.el8.noar 3.8 MB/s | 214 kB 00:00 2021-09-02T07:25:41.310 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: libcephsqlite-2:17.0.0-7401.g9366f375.el8.x86_64 6/7 2021-09-02T07:25:41.339 INFO:teuthology.orchestra.run.smithi175.stdout:(6/41): python3-lxml-4.2.3-2.el8.x86_64.rpm 27 MB/s | 1.5 MB 00:00 2021-09-02T07:25:41.374 INFO:teuthology.orchestra.run.smithi175.stdout:(7/41): python3-mako-1.0.6-13.el8.noarch.rpm 4.3 MB/s | 157 kB 00:00 2021-09-02T07:25:41.433 INFO:teuthology.orchestra.run.smithi175.stdout:(8/41): python3-markupsafe-0.23-19.el8.x86_64.r 655 kB/s | 39 kB 00:00 2021-09-02T07:25:41.439 INFO:teuthology.orchestra.run.smithi175.stdout:(9/41): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 19 MB/s | 103 kB 00:00 2021-09-02T07:25:41.473 INFO:teuthology.orchestra.run.smithi175.stdout:(10/41): python3-pytz-2017.2-9.el8.noarch.rpm 1.6 MB/s | 54 kB 00:00 2021-09-02T07:25:41.490 INFO:teuthology.orchestra.run.smithi175.stdout:(11/41): python3-webencodings-0.5.1-6.el8.noarc 1.6 MB/s | 27 kB 00:00 2021-09-02T07:25:41.515 INFO:teuthology.orchestra.run.smithi175.stdout:(12/41): python3-werkzeug-0.12.2-4.el8.noarch.r 18 MB/s | 457 kB 00:00 2021-09-02T07:25:41.524 INFO:teuthology.orchestra.run.smithi175.stdout:(13/41): python3-cffi-1.11.5-5.el8.x86_64.rpm 25 MB/s | 237 kB 00:00 2021-09-02T07:25:41.533 INFO:teuthology.orchestra.run.smithi175.stdout:(14/41): python3-chardet-3.0.4-7.el8.noarch.rpm 23 MB/s | 195 kB 00:00 2021-09-02T07:25:41.541 INFO:teuthology.orchestra.run.smithi175.stdout:(15/41): python3-cryptography-3.2.1-4.el8.x86_6 68 MB/s | 559 kB 00:00 2021-09-02T07:25:41.547 INFO:teuthology.orchestra.run.smithi175.stdout:(16/41): python3-idna-2.5-5.el8.noarch.rpm 17 MB/s | 97 kB 00:00 2021-09-02T07:25:41.549 INFO:teuthology.orchestra.run.smithi175.stdout:(17/41): python3-pycparser-2.14-14.el8.noarch.r 50 MB/s | 109 kB 00:00 2021-09-02T07:25:41.556 INFO:teuthology.orchestra.run.smithi175.stdout:(18/41): python3-pysocks-1.6.8-3.el8.noarch.rpm 5.5 MB/s | 34 kB 00:00 2021-09-02T07:25:41.558 INFO:teuthology.orchestra.run.smithi175.stdout:(19/41): python3-requests-2.20.0-2.1.el8_1.noar 53 MB/s | 123 kB 00:00 2021-09-02T07:25:41.561 INFO:teuthology.orchestra.run.smithi175.stdout:(20/41): python3-urllib3-1.24.2-5.el8.noarch.rp 58 MB/s | 177 kB 00:00 2021-09-02T07:25:41.575 INFO:teuthology.orchestra.run.smithi175.stdout:(21/41): ceph-mon-17.0.0-7401.g9366f375.el8.x86 4.1 MB/s | 3.9 MB 00:00 2021-09-02T07:25:41.677 INFO:teuthology.orchestra.run.smithi175.stdout:(22/41): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 380 kB/s | 44 kB 00:00 2021-09-02T07:25:41.719 INFO:teuthology.orchestra.run.smithi175.stdout:(23/41): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.3 MB/s | 185 kB 00:00 2021-09-02T07:25:41.760 INFO:teuthology.orchestra.run.smithi175.stdout:(24/41): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.0 MB/s | 173 kB 00:00 2021-09-02T07:25:41.776 INFO:teuthology.orchestra.run.smithi175.stdout:(25/41): python3-cherrypy-18.4.0-1.el8.noarch.r 6.5 MB/s | 384 kB 00:00 2021-09-02T07:25:41.789 INFO:teuthology.orchestra.run.smithi175.stdout:(26/41): python3-cssselect-0.9.2-10.el8.noarch. 1.4 MB/s | 40 kB 00:00 2021-09-02T07:25:41.804 INFO:teuthology.orchestra.run.smithi175.stdout:(27/41): python3-jaraco-6.2-6.el8.noarch.rpm 397 kB/s | 11 kB 00:00 2021-09-02T07:25:41.817 INFO:teuthology.orchestra.run.smithi175.stdout:(28/41): python3-jaraco-functools-2.0-4.el8.noa 627 kB/s | 18 kB 00:00 2021-09-02T07:25:41.833 INFO:teuthology.orchestra.run.smithi175.stdout:(29/41): python3-logutils-0.3.5-11.el8.noarch.r 1.7 MB/s | 49 kB 00:00 2021-09-02T07:25:41.847 INFO:teuthology.orchestra.run.smithi175.stdout:(30/41): python3-more-itertools-7.2.0-3.el8.noa 2.0 MB/s | 59 kB 00:00 2021-09-02T07:25:41.874 INFO:teuthology.orchestra.run.smithi175.stdout:(31/41): python3-portend-2.6-1.el8.noarch.rpm 559 kB/s | 16 kB 00:00 2021-09-02T07:25:41.890 INFO:teuthology.orchestra.run.smithi175.stdout:(32/41): python3-pecan-1.3.2-9.el8.noarch.rpm 5.0 MB/s | 283 kB 00:00 2021-09-02T07:25:41.903 INFO:teuthology.orchestra.run.smithi175.stdout:(33/41): python3-simplegeneric-0.8.1-17.el8.noa 634 kB/s | 19 kB 00:00 2021-09-02T07:25:41.917 INFO:teuthology.orchestra.run.smithi175.stdout:(34/41): python3-singledispatch-3.4.0.3-18.el8. 858 kB/s | 24 kB 00:00 2021-09-02T07:25:41.932 INFO:teuthology.orchestra.run.smithi175.stdout:(35/41): python3-tempora-1.14.1-5.el8.noarch.rp 1.0 MB/s | 29 kB 00:00 2021-09-02T07:25:41.945 INFO:teuthology.orchestra.run.smithi175.stdout:(36/41): python3-trustme-0.6.0-4.el8.noarch.rpm 975 kB/s | 27 kB 00:00 2021-09-02T07:25:41.985 INFO:teuthology.orchestra.run.smithi175.stdout:(37/41): python3-webob-1.8.5-1.el8.1.noarch.rpm 7.9 MB/s | 251 kB 00:00 2021-09-02T07:25:41.988 INFO:teuthology.orchestra.run.smithi175.stdout:(38/41): python3-waitress-1.2.1-2.el8.1.noarch. 4.2 MB/s | 241 kB 00:00 2021-09-02T07:25:42.005 INFO:teuthology.orchestra.run.smithi175.stdout:(39/41): python3-webtest-2.0.33-1.el8.noarch.rp 2.9 MB/s | 85 kB 00:00 2021-09-02T07:25:42.018 INFO:teuthology.orchestra.run.smithi175.stdout:(40/41): python3-zc-lockfile-2.0-2.el8.noarch.r 818 kB/s | 23 kB 00:00 2021-09-02T07:25:42.369 INFO:teuthology.orchestra.run.smithi175.stdout:(41/41): ceph-osd-17.0.0-7401.g9366f375.el8.x86 13 MB/s | 17 MB 00:01 2021-09-02T07:25:42.374 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:25:42.375 INFO:teuthology.orchestra.run.smithi175.stdout:Total 16 MB/s | 31 MB 00:01 2021-09-02T07:25:42.390 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2021-09-02T07:25:42.434 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2021-09-02T07:25:42.435 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2021-09-02T07:25:42.616 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : ceph-test-2:17.0.0-7401.g9366f375.el8.x86_64 7/7 2021-09-02T07:25:42.740 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2021-09-02T07:25:42.743 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2021-09-02T07:25:42.866 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: ceph-test-2:17.0.0-7401.g9366f375.el8.x86_64 7/7 2021-09-02T07:25:42.867 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : ceph-test-2:17.0.0-7401.g9366f375.el8.x86_64 1/7 2021-09-02T07:25:42.867 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : libcephsqlite-2:17.0.0-7401.g9366f375.el8.x86_64 2/7 2021-09-02T07:25:42.868 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : jq-1.5-12.el8.x86_64 3/7 2021-09-02T07:25:42.868 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-09-02T07:25:42.868 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 5/7 2021-09-02T07:25:42.869 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : libxslt-1.1.32-6.el8.x86_64 6/7 2021-09-02T07:25:43.271 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 7/7 2021-09-02T07:25:43.271 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:25:43.272 INFO:teuthology.orchestra.run.smithi197.stdout:Installed: 2021-09-02T07:25:43.272 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-test-2:17.0.0-7401.g9366f375.el8.x86_64 jq-1.5-12.el8.x86_64 2021-09-02T07:25:43.272 INFO:teuthology.orchestra.run.smithi197.stdout: libcephsqlite-2:17.0.0-7401.g9366f375.el8.x86_64 libxslt-1.1.32-6.el8.x86_64 2021-09-02T07:25:43.272 INFO:teuthology.orchestra.run.smithi197.stdout: oniguruma-6.8.2-2.el8.x86_64 socat-1.7.3.3-2.el8.x86_64 2021-09-02T07:25:43.272 INFO:teuthology.orchestra.run.smithi197.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-09-02T07:25:43.273 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:25:43.274 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:25:43.407 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install ceph 2021-09-02T07:25:43.852 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:00:50 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:25:44.189 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2021-09-02T07:25:44.405 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/41 2021-09-02T07:25:44.634 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/41 2021-09-02T07:25:44.789 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-idna-2.5-5.el8.noarch 3/41 2021-09-02T07:25:44.793 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:25:44.798 INFO:teuthology.orchestra.run.smithi197.stdout:======================================================================================= 2021-09-02T07:25:44.798 INFO:teuthology.orchestra.run.smithi197.stdout: Package Arch Version Repository Size 2021-09-02T07:25:44.798 INFO:teuthology.orchestra.run.smithi197.stdout:======================================================================================= 2021-09-02T07:25:44.798 INFO:teuthology.orchestra.run.smithi197.stdout:Installing: 2021-09-02T07:25:44.799 INFO:teuthology.orchestra.run.smithi197.stdout: ceph x86_64 2:17.0.0-7401.g9366f375.el8 ceph 6.4 k 2021-09-02T07:25:44.799 INFO:teuthology.orchestra.run.smithi197.stdout:Installing dependencies: 2021-09-02T07:25:44.799 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mds x86_64 2:17.0.0-7401.g9366f375.el8 ceph 2.3 M 2021-09-02T07:25:44.799 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mgr x86_64 2:17.0.0-7401.g9366f375.el8 ceph 1.5 M 2021-09-02T07:25:44.800 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mgr-modules-core noarch 2:17.0.0-7401.g9366f375.el8 ceph-noarch 214 k 2021-09-02T07:25:44.800 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mon x86_64 2:17.0.0-7401.g9366f375.el8 ceph 3.9 M 2021-09-02T07:25:44.800 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-osd x86_64 2:17.0.0-7401.g9366f375.el8 ceph 17 M 2021-09-02T07:25:44.800 INFO:teuthology.orchestra.run.smithi197.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-09-02T07:25:44.801 INFO:teuthology.orchestra.run.smithi197.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-09-02T07:25:44.801 INFO:teuthology.orchestra.run.smithi197.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-09-02T07:25:44.801 INFO:teuthology.orchestra.run.smithi197.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-09-02T07:25:44.801 INFO:teuthology.orchestra.run.smithi197.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-09-02T07:25:44.802 INFO:teuthology.orchestra.run.smithi197.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-09-02T07:25:44.802 INFO:teuthology.orchestra.run.smithi197.stdout: python3-cryptography x86_64 3.2.1-4.el8 CentOS-Base 559 k 2021-09-02T07:25:44.803 INFO:teuthology.orchestra.run.smithi197.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-09-02T07:25:44.803 INFO:teuthology.orchestra.run.smithi197.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-09-02T07:25:44.803 INFO:teuthology.orchestra.run.smithi197.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-09-02T07:25:44.804 INFO:teuthology.orchestra.run.smithi197.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-09-02T07:25:44.804 INFO:teuthology.orchestra.run.smithi197.stdout: python3-lxml x86_64 4.2.3-2.el8 CentOS-AppStream 1.5 M 2021-09-02T07:25:44.804 INFO:teuthology.orchestra.run.smithi197.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-09-02T07:25:44.804 INFO:teuthology.orchestra.run.smithi197.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-09-02T07:25:44.805 INFO:teuthology.orchestra.run.smithi197.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-09-02T07:25:44.805 INFO:teuthology.orchestra.run.smithi197.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-09-02T07:25:44.805 INFO:teuthology.orchestra.run.smithi197.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-09-02T07:25:44.805 INFO:teuthology.orchestra.run.smithi197.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2021-09-02T07:25:44.806 INFO:teuthology.orchestra.run.smithi197.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-09-02T07:25:44.806 INFO:teuthology.orchestra.run.smithi197.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-09-02T07:25:44.806 INFO:teuthology.orchestra.run.smithi197.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-09-02T07:25:44.806 INFO:teuthology.orchestra.run.smithi197.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-09-02T07:25:44.807 INFO:teuthology.orchestra.run.smithi197.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-09-02T07:25:44.807 INFO:teuthology.orchestra.run.smithi197.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-09-02T07:25:44.807 INFO:teuthology.orchestra.run.smithi197.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-09-02T07:25:44.807 INFO:teuthology.orchestra.run.smithi197.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-09-02T07:25:44.807 INFO:teuthology.orchestra.run.smithi197.stdout: python3-urllib3 noarch 1.24.2-5.el8 CentOS-Base 177 k 2021-09-02T07:25:44.808 INFO:teuthology.orchestra.run.smithi197.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-09-02T07:25:44.808 INFO:teuthology.orchestra.run.smithi197.stdout: python3-webencodings noarch 0.5.1-6.el8 CentOS-AppStream 27 k 2021-09-02T07:25:44.808 INFO:teuthology.orchestra.run.smithi197.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-09-02T07:25:44.808 INFO:teuthology.orchestra.run.smithi197.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-09-02T07:25:44.808 INFO:teuthology.orchestra.run.smithi197.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-09-02T07:25:44.809 INFO:teuthology.orchestra.run.smithi197.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-09-02T07:25:44.809 INFO:teuthology.orchestra.run.smithi197.stdout:Installing weak dependencies: 2021-09-02T07:25:44.809 INFO:teuthology.orchestra.run.smithi197.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-09-02T07:25:44.809 INFO:teuthology.orchestra.run.smithi197.stdout: python3-html5lib noarch 1:0.999999999-6.el8 CentOS-AppStream 214 k 2021-09-02T07:25:44.810 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:25:44.810 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction Summary 2021-09-02T07:25:44.811 INFO:teuthology.orchestra.run.smithi197.stdout:======================================================================================= 2021-09-02T07:25:44.811 INFO:teuthology.orchestra.run.smithi197.stdout:Install 41 Packages 2021-09-02T07:25:44.811 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:25:44.812 INFO:teuthology.orchestra.run.smithi197.stdout:Total download size: 31 M 2021-09-02T07:25:44.812 INFO:teuthology.orchestra.run.smithi197.stdout:Installed size: 121 M 2021-09-02T07:25:44.813 INFO:teuthology.orchestra.run.smithi197.stdout:Downloading Packages: 2021-09-02T07:25:44.923 INFO:teuthology.orchestra.run.smithi197.stdout:(1/41): ceph-17.0.0-7401.g9366f375.el8.x86_64.r 54 kB/s | 6.4 kB 00:00 2021-09-02T07:25:45.032 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 4/41 2021-09-02T07:25:45.107 INFO:teuthology.orchestra.run.smithi197.stdout:(2/41): ceph-mgr-17.0.0-7401.g9366f375.el8.x86_ 5.0 MB/s | 1.5 MB 00:00 2021-09-02T07:25:45.124 INFO:teuthology.orchestra.run.smithi197.stdout:(3/41): ceph-mds-17.0.0-7401.g9366f375.el8.x86_ 7.0 MB/s | 2.3 MB 00:00 2021-09-02T07:25:45.158 INFO:teuthology.orchestra.run.smithi197.stdout:(4/41): ceph-mgr-modules-core-17.0.0-7401.g9366 6.3 MB/s | 214 kB 00:00 2021-09-02T07:25:45.190 INFO:teuthology.orchestra.run.smithi197.stdout:(5/41): python3-html5lib-0.999999999-6.el8.noar 6.4 MB/s | 214 kB 00:00 2021-09-02T07:25:45.211 INFO:teuthology.orchestra.run.smithi197.stdout:(6/41): python3-lxml-4.2.3-2.el8.x86_64.rpm 68 MB/s | 1.5 MB 00:00 2021-09-02T07:25:45.212 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 5/41 2021-09-02T07:25:45.214 INFO:teuthology.orchestra.run.smithi197.stdout:(7/41): python3-mako-1.0.6-13.el8.noarch.rpm 56 MB/s | 157 kB 00:00 2021-09-02T07:25:45.215 INFO:teuthology.orchestra.run.smithi197.stdout:(8/41): python3-markupsafe-0.23-19.el8.x86_64.r 29 MB/s | 39 kB 00:00 2021-09-02T07:25:45.218 INFO:teuthology.orchestra.run.smithi197.stdout:(9/41): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 55 MB/s | 103 kB 00:00 2021-09-02T07:25:45.219 INFO:teuthology.orchestra.run.smithi197.stdout:(10/41): python3-pytz-2017.2-9.el8.noarch.rpm 36 MB/s | 54 kB 00:00 2021-09-02T07:25:45.220 INFO:teuthology.orchestra.run.smithi197.stdout:(11/41): python3-webencodings-0.5.1-6.el8.noarc 23 MB/s | 27 kB 00:00 2021-09-02T07:25:45.231 INFO:teuthology.orchestra.run.smithi197.stdout:(12/41): python3-werkzeug-0.12.2-4.el8.noarch.r 75 MB/s | 457 kB 00:00 2021-09-02T07:25:45.231 INFO:teuthology.orchestra.run.smithi197.stdout:(13/41): python3-cffi-1.11.5-5.el8.x86_64.rpm 67 MB/s | 237 kB 00:00 2021-09-02T07:25:45.233 INFO:teuthology.orchestra.run.smithi197.stdout:(14/41): python3-chardet-3.0.4-7.el8.noarch.rpm 65 MB/s | 195 kB 00:00 2021-09-02T07:25:45.241 INFO:teuthology.orchestra.run.smithi197.stdout:(15/41): python3-cryptography-3.2.1-4.el8.x86_6 77 MB/s | 559 kB 00:00 2021-09-02T07:25:45.243 INFO:teuthology.orchestra.run.smithi197.stdout:(16/41): python3-idna-2.5-5.el8.noarch.rpm 49 MB/s | 97 kB 00:00 2021-09-02T07:25:45.246 INFO:teuthology.orchestra.run.smithi197.stdout:(17/41): python3-pycparser-2.14-14.el8.noarch.r 54 MB/s | 109 kB 00:00 2021-09-02T07:25:45.247 INFO:teuthology.orchestra.run.smithi197.stdout:(18/41): python3-pysocks-1.6.8-3.el8.noarch.rpm 27 MB/s | 34 kB 00:00 2021-09-02T07:25:45.250 INFO:teuthology.orchestra.run.smithi197.stdout:(19/41): python3-requests-2.20.0-2.1.el8_1.noar 57 MB/s | 123 kB 00:00 2021-09-02T07:25:45.263 INFO:teuthology.orchestra.run.smithi197.stdout:(20/41): python3-urllib3-1.24.2-5.el8.noarch.rp 63 MB/s | 177 kB 00:00 2021-09-02T07:25:45.350 INFO:teuthology.orchestra.run.smithi197.stdout:(21/41): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 445 kB/s | 44 kB 00:00 2021-09-02T07:25:45.360 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/41 2021-09-02T07:25:45.423 INFO:teuthology.orchestra.run.smithi197.stdout:(22/41): python3-beautifulsoup4-4.6.3-2.el8.1.n 2.5 MB/s | 185 kB 00:00 2021-09-02T07:25:45.450 INFO:teuthology.orchestra.run.smithi197.stdout:(23/41): python3-cheroot-8.5.2-1.el8.noarch.rpm 6.4 MB/s | 173 kB 00:00 2021-09-02T07:25:45.483 INFO:teuthology.orchestra.run.smithi197.stdout:(24/41): ceph-mon-17.0.0-7401.g9366f375.el8.x86 6.9 MB/s | 3.9 MB 00:00 2021-09-02T07:25:45.501 INFO:teuthology.orchestra.run.smithi197.stdout:(25/41): python3-cherrypy-18.4.0-1.el8.noarch.r 7.5 MB/s | 384 kB 00:00 2021-09-02T07:25:45.507 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/41 2021-09-02T07:25:45.526 INFO:teuthology.orchestra.run.smithi197.stdout:(26/41): python3-jaraco-6.2-6.el8.noarch.rpm 455 kB/s | 11 kB 00:00 2021-09-02T07:25:45.550 INFO:teuthology.orchestra.run.smithi197.stdout:(27/41): python3-jaraco-functools-2.0-4.el8.noa 716 kB/s | 18 kB 00:00 2021-09-02T07:25:45.566 INFO:teuthology.orchestra.run.smithi197.stdout:(28/41): python3-cssselect-0.9.2-10.el8.noarch. 482 kB/s | 40 kB 00:00 2021-09-02T07:25:45.575 INFO:teuthology.orchestra.run.smithi197.stdout:(29/41): python3-logutils-0.3.5-11.el8.noarch.r 1.9 MB/s | 49 kB 00:00 2021-09-02T07:25:45.621 INFO:teuthology.orchestra.run.smithi197.stdout:(30/41): python3-more-itertools-7.2.0-3.el8.noa 1.0 MB/s | 59 kB 00:00 2021-09-02T07:25:45.624 INFO:teuthology.orchestra.run.smithi197.stdout:(31/41): python3-pecan-1.3.2-9.el8.noarch.rpm 5.6 MB/s | 283 kB 00:00 2021-09-02T07:25:45.650 INFO:teuthology.orchestra.run.smithi197.stdout:(32/41): python3-simplegeneric-0.8.1-17.el8.noa 745 kB/s | 19 kB 00:00 2021-09-02T07:25:45.650 INFO:teuthology.orchestra.run.smithi197.stdout:(33/41): python3-portend-2.6-1.el8.noarch.rpm 547 kB/s | 16 kB 00:00 2021-09-02T07:25:45.674 INFO:teuthology.orchestra.run.smithi197.stdout:(34/41): python3-singledispatch-3.4.0.3-18.el8. 970 kB/s | 24 kB 00:00 2021-09-02T07:25:45.677 INFO:teuthology.orchestra.run.smithi197.stdout:(35/41): python3-tempora-1.14.1-5.el8.noarch.rp 1.0 MB/s | 29 kB 00:00 2021-09-02T07:25:45.699 INFO:teuthology.orchestra.run.smithi197.stdout:(36/41): python3-trustme-0.6.0-4.el8.noarch.rpm 1.1 MB/s | 27 kB 00:00 2021-09-02T07:25:45.708 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/41 2021-09-02T07:25:45.727 INFO:teuthology.orchestra.run.smithi197.stdout:(37/41): python3-webob-1.8.5-1.el8.1.noarch.rpm 9.0 MB/s | 251 kB 00:00 2021-09-02T07:25:45.734 INFO:teuthology.orchestra.run.smithi197.stdout:(38/41): python3-waitress-1.2.1-2.el8.1.noarch. 4.1 MB/s | 241 kB 00:00 2021-09-02T07:25:45.752 INFO:teuthology.orchestra.run.smithi197.stdout:(39/41): python3-webtest-2.0.33-1.el8.noarch.rp 3.3 MB/s | 85 kB 00:00 2021-09-02T07:25:45.762 INFO:teuthology.orchestra.run.smithi197.stdout:(40/41): python3-zc-lockfile-2.0-2.el8.noarch.r 824 kB/s | 23 kB 00:00 2021-09-02T07:25:45.847 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/41 2021-09-02T07:25:45.924 INFO:teuthology.orchestra.run.smithi197.stdout:(41/41): ceph-osd-17.0.0-7401.g9366f375.el8.x86 21 MB/s | 17 MB 00:00 2021-09-02T07:25:45.926 INFO:teuthology.orchestra.run.smithi197.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:25:45.926 INFO:teuthology.orchestra.run.smithi197.stdout:Total 27 MB/s | 31 MB 00:01 2021-09-02T07:25:45.946 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction check 2021-09-02T07:25:45.980 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction check succeeded. 2021-09-02T07:25:45.980 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction test 2021-09-02T07:25:45.991 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/41 2021-09-02T07:25:46.137 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 11/41 2021-09-02T07:25:46.282 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction test succeeded. 2021-09-02T07:25:46.285 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction 2021-09-02T07:25:46.333 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 12/41 2021-09-02T07:25:46.577 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-urllib3-1.24.2-5.el8.noarch 13/41 2021-09-02T07:25:46.801 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 14/41 2021-09-02T07:25:47.170 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 15/41 2021-09-02T07:25:47.451 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cryptography-3.2.1-4.el8.x86_64 16/41 2021-09-02T07:25:47.638 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 17/41 2021-09-02T07:25:47.772 INFO:teuthology.orchestra.run.smithi197.stdout: Preparing : 1/1 2021-09-02T07:25:47.840 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 18/41 2021-09-02T07:25:48.000 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/41 2021-09-02T07:25:48.023 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 19/41 2021-09-02T07:25:48.202 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/41 2021-09-02T07:25:48.254 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 20/41 2021-09-02T07:25:48.382 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-idna-2.5-5.el8.noarch 3/41 2021-09-02T07:25:48.443 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 21/41 2021-09-02T07:25:48.599 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 4/41 2021-09-02T07:25:48.674 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 22/41 2021-09-02T07:25:48.816 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 5/41 2021-09-02T07:25:48.874 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/41 2021-09-02T07:25:48.962 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/41 2021-09-02T07:25:49.108 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 24/41 2021-09-02T07:25:49.111 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/41 2021-09-02T07:25:49.294 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/41 2021-09-02T07:25:49.335 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 25/41 2021-09-02T07:25:49.431 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/41 2021-09-02T07:25:49.580 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/41 2021-09-02T07:25:49.714 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 26/41 2021-09-02T07:25:49.717 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 11/41 2021-09-02T07:25:49.959 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-lxml-4.2.3-2.el8.x86_64 27/41 2021-09-02T07:25:49.960 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 12/41 2021-09-02T07:25:50.176 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-urllib3-1.24.2-5.el8.noarch 13/41 2021-09-02T07:25:50.186 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 28/41 2021-09-02T07:25:50.340 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 29/41 2021-09-02T07:25:50.400 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 14/41 2021-09-02T07:25:50.501 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 30/41 2021-09-02T07:25:50.733 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-portend-2.6-1.el8.noarch 31/41 2021-09-02T07:25:50.757 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 15/41 2021-09-02T07:25:50.934 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 32/41 2021-09-02T07:25:51.003 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-cryptography-3.2.1-4.el8.x86_64 16/41 2021-09-02T07:25:51.133 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 33/41 2021-09-02T07:25:51.166 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 17/41 2021-09-02T07:25:51.394 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 18/41 2021-09-02T07:25:51.453 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 34/41 2021-09-02T07:25:51.592 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 19/41 2021-09-02T07:25:51.771 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 35/41 2021-09-02T07:25:51.825 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 20/41 2021-09-02T07:25:52.029 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 21/41 2021-09-02T07:25:52.141 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-7401.g9366f375.el8. 36/41 2021-09-02T07:25:52.303 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 22/41 2021-09-02T07:25:52.322 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-2:17.0.0-7401.g9366f375.el8.x86_64 37/41 2021-09-02T07:25:52.494 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/41 2021-09-02T07:25:52.705 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 24/41 2021-09-02T07:25:52.904 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 25/41 2021-09-02T07:25:53.284 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 26/41 2021-09-02T07:25:53.494 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-lxml-4.2.3-2.el8.x86_64 27/41 2021-09-02T07:25:53.697 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 28/41 2021-09-02T07:25:53.877 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 29/41 2021-09-02T07:25:53.929 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-2:17.0.0-7401.g9366f375.el8.x86_64 37/41 2021-09-02T07:25:54.037 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 30/41 2021-09-02T07:25:54.212 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-osd-2:17.0.0-7401.g9366f375.el8.x86_64 38/41 2021-09-02T07:25:54.318 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-portend-2.6-1.el8.noarch 31/41 2021-09-02T07:25:54.503 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 32/41 2021-09-02T07:25:54.614 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-osd-2:17.0.0-7401.g9366f375.el8.x86_64 38/41 2021-09-02T07:25:54.755 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 33/41 2021-09-02T07:25:54.797 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mon-2:17.0.0-7401.g9366f375.el8.x86_64 39/41 2021-09-02T07:25:55.009 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mon-2:17.0.0-7401.g9366f375.el8.x86_64 39/41 2021-09-02T07:25:55.076 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 34/41 2021-09-02T07:25:55.178 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mds-2:17.0.0-7401.g9366f375.el8.x86_64 40/41 2021-09-02T07:25:55.194 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mds-2:17.0.0-7401.g9366f375.el8.x86_64 40/41 2021-09-02T07:25:55.328 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-2:17.0.0-7401.g9366f375.el8.x86_64 41/41 2021-09-02T07:25:55.415 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 35/41 2021-09-02T07:25:55.721 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-2:17.0.0-7401.g9366f375.el8.x86_64 41/41 2021-09-02T07:25:55.722 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-2:17.0.0-7401.g9366f375.el8.x86_64 1/41 2021-09-02T07:25:55.722 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mds-2:17.0.0-7401.g9366f375.el8.x86_64 2/41 2021-09-02T07:25:55.722 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-2:17.0.0-7401.g9366f375.el8.x86_64 3/41 2021-09-02T07:25:55.723 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mon-2:17.0.0-7401.g9366f375.el8.x86_64 4/41 2021-09-02T07:25:55.723 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-osd-2:17.0.0-7401.g9366f375.el8.x86_64 5/41 2021-09-02T07:25:55.723 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-7401.g9366f375.el8. 6/41 2021-09-02T07:25:55.723 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/41 2021-09-02T07:25:55.723 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-lxml-4.2.3-2.el8.x86_64 8/41 2021-09-02T07:25:55.724 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/41 2021-09-02T07:25:55.724 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/41 2021-09-02T07:25:55.724 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 11/41 2021-09-02T07:25:55.724 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 12/41 2021-09-02T07:25:55.725 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 13/41 2021-09-02T07:25:55.725 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 14/41 2021-09-02T07:25:55.725 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 15/41 2021-09-02T07:25:55.725 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 16/41 2021-09-02T07:25:55.726 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cryptography-3.2.1-4.el8.x86_64 17/41 2021-09-02T07:25:55.726 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-idna-2.5-5.el8.noarch 18/41 2021-09-02T07:25:55.726 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 19/41 2021-09-02T07:25:55.726 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 20/41 2021-09-02T07:25:55.727 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 21/41 2021-09-02T07:25:55.727 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-urllib3-1.24.2-5.el8.noarch 22/41 2021-09-02T07:25:55.727 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 23/41 2021-09-02T07:25:55.727 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/41 2021-09-02T07:25:55.727 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 25/41 2021-09-02T07:25:55.728 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 26/41 2021-09-02T07:25:55.728 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 27/41 2021-09-02T07:25:55.728 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 28/41 2021-09-02T07:25:55.730 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 29/41 2021-09-02T07:25:55.730 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 30/41 2021-09-02T07:25:55.730 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 31/41 2021-09-02T07:25:55.731 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 32/41 2021-09-02T07:25:55.731 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-portend-2.6-1.el8.noarch 33/41 2021-09-02T07:25:55.731 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 34/41 2021-09-02T07:25:55.731 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 35/41 2021-09-02T07:25:55.731 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 36/41 2021-09-02T07:25:55.732 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 37/41 2021-09-02T07:25:55.732 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 38/41 2021-09-02T07:25:55.732 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 39/41 2021-09-02T07:25:55.732 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 40/41 2021-09-02T07:25:55.787 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-7401.g9366f375.el8. 36/41 2021-09-02T07:25:55.935 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : ceph-mgr-2:17.0.0-7401.g9366f375.el8.x86_64 37/41 2021-09-02T07:25:56.670 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 41/41 2021-09-02T07:25:56.670 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:25:56.670 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2021-09-02T07:25:56.670 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:56.671 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mds-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:56.671 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:56.671 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-modules-core-2:17.0.0-7401.g9366f375.el8.noarch 2021-09-02T07:25:56.671 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mon-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:56.672 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-osd-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:25:56.672 INFO:teuthology.orchestra.run.smithi175.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-09-02T07:25:56.672 INFO:teuthology.orchestra.run.smithi175.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-09-02T07:25:56.672 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-09-02T07:25:56.673 INFO:teuthology.orchestra.run.smithi175.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-09-02T07:25:56.673 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-09-02T07:25:56.673 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-09-02T07:25:56.673 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cryptography-3.2.1-4.el8.x86_64 2021-09-02T07:25:56.673 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-09-02T07:25:56.674 INFO:teuthology.orchestra.run.smithi175.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-09-02T07:25:56.674 INFO:teuthology.orchestra.run.smithi175.stdout: python3-idna-2.5-5.el8.noarch 2021-09-02T07:25:56.674 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-6.2-6.el8.noarch 2021-09-02T07:25:56.674 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-09-02T07:25:56.675 INFO:teuthology.orchestra.run.smithi175.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-09-02T07:25:56.675 INFO:teuthology.orchestra.run.smithi175.stdout: python3-lxml-4.2.3-2.el8.x86_64 2021-09-02T07:25:56.675 INFO:teuthology.orchestra.run.smithi175.stdout: python3-mako-1.0.6-13.el8.noarch 2021-09-02T07:25:56.675 INFO:teuthology.orchestra.run.smithi175.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-09-02T07:25:56.676 INFO:teuthology.orchestra.run.smithi175.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-09-02T07:25:56.676 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-09-02T07:25:56.676 INFO:teuthology.orchestra.run.smithi175.stdout: python3-portend-2.6-1.el8.noarch 2021-09-02T07:25:56.676 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-09-02T07:25:56.676 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pycparser-2.14-14.el8.noarch 2021-09-02T07:25:56.677 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-09-02T07:25:56.677 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pytz-2017.2-9.el8.noarch 2021-09-02T07:25:56.677 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-09-02T07:25:56.677 INFO:teuthology.orchestra.run.smithi175.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-09-02T07:25:56.677 INFO:teuthology.orchestra.run.smithi175.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-09-02T07:25:56.678 INFO:teuthology.orchestra.run.smithi175.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-09-02T07:25:56.678 INFO:teuthology.orchestra.run.smithi175.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-09-02T07:25:56.678 INFO:teuthology.orchestra.run.smithi175.stdout: python3-urllib3-1.24.2-5.el8.noarch 2021-09-02T07:25:56.678 INFO:teuthology.orchestra.run.smithi175.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-09-02T07:25:56.679 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-09-02T07:25:56.679 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-09-02T07:25:56.679 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-09-02T07:25:56.680 INFO:teuthology.orchestra.run.smithi175.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-09-02T07:25:56.680 INFO:teuthology.orchestra.run.smithi175.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-09-02T07:25:56.681 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:25:56.681 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:25:56.856 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-base 2021-09-02T07:25:57.311 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:04 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:25:57.547 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: ceph-mgr-2:17.0.0-7401.g9366f375.el8.x86_64 37/41 2021-09-02T07:25:57.823 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : ceph-osd-2:17.0.0-7401.g9366f375.el8.x86_64 38/41 2021-09-02T07:25:58.020 INFO:teuthology.orchestra.run.smithi175.stdout:Package ceph-base-2:17.0.0-7401.g9366f375.el8.x86_64 is already installed. 2021-09-02T07:25:58.246 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: ceph-osd-2:17.0.0-7401.g9366f375.el8.x86_64 38/41 2021-09-02T07:25:58.269 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:25:58.270 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2021-09-02T07:25:58.271 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:25:58.332 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install cephadm 2021-09-02T07:25:58.490 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : ceph-mon-2:17.0.0-7401.g9366f375.el8.x86_64 39/41 2021-09-02T07:25:58.704 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: ceph-mon-2:17.0.0-7401.g9366f375.el8.x86_64 39/41 2021-09-02T07:25:58.776 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:05 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:25:58.865 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : ceph-mds-2:17.0.0-7401.g9366f375.el8.x86_64 40/41 2021-09-02T07:25:58.881 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: ceph-mds-2:17.0.0-7401.g9366f375.el8.x86_64 40/41 2021-09-02T07:25:59.030 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : ceph-2:17.0.0-7401.g9366f375.el8.x86_64 41/41 2021-09-02T07:25:59.401 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: ceph-2:17.0.0-7401.g9366f375.el8.x86_64 41/41 2021-09-02T07:25:59.402 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : ceph-2:17.0.0-7401.g9366f375.el8.x86_64 1/41 2021-09-02T07:25:59.402 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : ceph-mds-2:17.0.0-7401.g9366f375.el8.x86_64 2/41 2021-09-02T07:25:59.402 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : ceph-mgr-2:17.0.0-7401.g9366f375.el8.x86_64 3/41 2021-09-02T07:25:59.402 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : ceph-mon-2:17.0.0-7401.g9366f375.el8.x86_64 4/41 2021-09-02T07:25:59.402 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : ceph-osd-2:17.0.0-7401.g9366f375.el8.x86_64 5/41 2021-09-02T07:25:59.403 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-7401.g9366f375.el8. 6/41 2021-09-02T07:25:59.403 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/41 2021-09-02T07:25:59.403 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-lxml-4.2.3-2.el8.x86_64 8/41 2021-09-02T07:25:59.404 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/41 2021-09-02T07:25:59.404 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/41 2021-09-02T07:25:59.404 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 11/41 2021-09-02T07:25:59.404 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 12/41 2021-09-02T07:25:59.405 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 13/41 2021-09-02T07:25:59.405 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 14/41 2021-09-02T07:25:59.405 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 15/41 2021-09-02T07:25:59.405 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 16/41 2021-09-02T07:25:59.406 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-cryptography-3.2.1-4.el8.x86_64 17/41 2021-09-02T07:25:59.406 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-idna-2.5-5.el8.noarch 18/41 2021-09-02T07:25:59.406 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 19/41 2021-09-02T07:25:59.406 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 20/41 2021-09-02T07:25:59.407 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 21/41 2021-09-02T07:25:59.407 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-urllib3-1.24.2-5.el8.noarch 22/41 2021-09-02T07:25:59.407 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 23/41 2021-09-02T07:25:59.407 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/41 2021-09-02T07:25:59.408 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 25/41 2021-09-02T07:25:59.408 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 26/41 2021-09-02T07:25:59.408 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 27/41 2021-09-02T07:25:59.410 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 28/41 2021-09-02T07:25:59.410 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 29/41 2021-09-02T07:25:59.410 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 30/41 2021-09-02T07:25:59.411 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 31/41 2021-09-02T07:25:59.411 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 32/41 2021-09-02T07:25:59.411 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-portend-2.6-1.el8.noarch 33/41 2021-09-02T07:25:59.411 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 34/41 2021-09-02T07:25:59.411 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 35/41 2021-09-02T07:25:59.412 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 36/41 2021-09-02T07:25:59.412 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 37/41 2021-09-02T07:25:59.412 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 38/41 2021-09-02T07:25:59.412 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 39/41 2021-09-02T07:25:59.413 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 40/41 2021-09-02T07:25:59.728 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:25:59.730 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:25:59.730 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2021-09-02T07:25:59.730 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:25:59.731 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2021-09-02T07:25:59.731 INFO:teuthology.orchestra.run.smithi175.stdout: cephadm noarch 2:17.0.0-7401.g9366f375.el8 ceph-noarch 69 k 2021-09-02T07:25:59.731 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:25:59.732 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2021-09-02T07:25:59.732 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:25:59.732 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2021-09-02T07:25:59.732 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:25:59.733 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 69 k 2021-09-02T07:25:59.733 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 309 k 2021-09-02T07:25:59.734 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2021-09-02T07:25:59.864 INFO:teuthology.orchestra.run.smithi175.stdout:cephadm-17.0.0-7401.g9366f375.el8.noarch.rpm 523 kB/s | 69 kB 00:00 2021-09-02T07:25:59.865 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:25:59.865 INFO:teuthology.orchestra.run.smithi175.stdout:Total 518 kB/s | 69 kB 00:00 2021-09-02T07:25:59.865 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2021-09-02T07:25:59.871 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2021-09-02T07:25:59.875 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2021-09-02T07:25:59.875 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2021-09-02T07:25:59.876 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2021-09-02T07:25:59.917 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2021-09-02T07:26:00.268 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 41/41 2021-09-02T07:26:00.269 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:00.269 INFO:teuthology.orchestra.run.smithi197.stdout:Installed: 2021-09-02T07:26:00.269 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:26:00.270 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mds-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:26:00.270 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mgr-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:26:00.270 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mgr-modules-core-2:17.0.0-7401.g9366f375.el8.noarch 2021-09-02T07:26:00.271 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mon-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:26:00.271 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-osd-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:26:00.272 INFO:teuthology.orchestra.run.smithi197.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-09-02T07:26:00.272 INFO:teuthology.orchestra.run.smithi197.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-09-02T07:26:00.272 INFO:teuthology.orchestra.run.smithi197.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-09-02T07:26:00.272 INFO:teuthology.orchestra.run.smithi197.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-09-02T07:26:00.273 INFO:teuthology.orchestra.run.smithi197.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-09-02T07:26:00.273 INFO:teuthology.orchestra.run.smithi197.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-09-02T07:26:00.273 INFO:teuthology.orchestra.run.smithi197.stdout: python3-cryptography-3.2.1-4.el8.x86_64 2021-09-02T07:26:00.273 INFO:teuthology.orchestra.run.smithi197.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-09-02T07:26:00.274 INFO:teuthology.orchestra.run.smithi197.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-09-02T07:26:00.274 INFO:teuthology.orchestra.run.smithi197.stdout: python3-idna-2.5-5.el8.noarch 2021-09-02T07:26:00.275 INFO:teuthology.orchestra.run.smithi197.stdout: python3-jaraco-6.2-6.el8.noarch 2021-09-02T07:26:00.275 INFO:teuthology.orchestra.run.smithi197.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-09-02T07:26:00.276 INFO:teuthology.orchestra.run.smithi197.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-09-02T07:26:00.276 INFO:teuthology.orchestra.run.smithi197.stdout: python3-lxml-4.2.3-2.el8.x86_64 2021-09-02T07:26:00.276 INFO:teuthology.orchestra.run.smithi197.stdout: python3-mako-1.0.6-13.el8.noarch 2021-09-02T07:26:00.277 INFO:teuthology.orchestra.run.smithi197.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-09-02T07:26:00.277 INFO:teuthology.orchestra.run.smithi197.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-09-02T07:26:00.277 INFO:teuthology.orchestra.run.smithi197.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-09-02T07:26:00.277 INFO:teuthology.orchestra.run.smithi197.stdout: python3-portend-2.6-1.el8.noarch 2021-09-02T07:26:00.278 INFO:teuthology.orchestra.run.smithi197.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-09-02T07:26:00.278 INFO:teuthology.orchestra.run.smithi197.stdout: python3-pycparser-2.14-14.el8.noarch 2021-09-02T07:26:00.278 INFO:teuthology.orchestra.run.smithi197.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-09-02T07:26:00.279 INFO:teuthology.orchestra.run.smithi197.stdout: python3-pytz-2017.2-9.el8.noarch 2021-09-02T07:26:00.280 INFO:teuthology.orchestra.run.smithi197.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-09-02T07:26:00.280 INFO:teuthology.orchestra.run.smithi197.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-09-02T07:26:00.281 INFO:teuthology.orchestra.run.smithi197.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-09-02T07:26:00.281 INFO:teuthology.orchestra.run.smithi197.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-09-02T07:26:00.281 INFO:teuthology.orchestra.run.smithi197.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-09-02T07:26:00.282 INFO:teuthology.orchestra.run.smithi197.stdout: python3-urllib3-1.24.2-5.el8.noarch 2021-09-02T07:26:00.282 INFO:teuthology.orchestra.run.smithi197.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-09-02T07:26:00.282 INFO:teuthology.orchestra.run.smithi197.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-09-02T07:26:00.283 INFO:teuthology.orchestra.run.smithi197.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-09-02T07:26:00.283 INFO:teuthology.orchestra.run.smithi197.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-09-02T07:26:00.284 INFO:teuthology.orchestra.run.smithi197.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-09-02T07:26:00.284 INFO:teuthology.orchestra.run.smithi197.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-09-02T07:26:00.285 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:00.286 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:26:00.456 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install ceph-base 2021-09-02T07:26:00.909 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:01:07 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:01.588 INFO:teuthology.orchestra.run.smithi197.stdout:Package ceph-base-2:17.0.0-7401.g9366f375.el8.x86_64 is already installed. 2021-09-02T07:26:01.837 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:26:01.838 INFO:teuthology.orchestra.run.smithi197.stdout:Nothing to do. 2021-09-02T07:26:01.838 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:26:01.884 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install cephadm 2021-09-02T07:26:02.072 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: cephadm-2:17.0.0-7401.g9366f375.el8.noarch 1/1 2021-09-02T07:26:02.292 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : cephadm-2:17.0.0-7401.g9366f375.el8.noarch 1/1 2021-09-02T07:26:02.327 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:01:09 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:02.471 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: cephadm-2:17.0.0-7401.g9366f375.el8.noarch 1/1 2021-09-02T07:26:02.778 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : cephadm-2:17.0.0-7401.g9366f375.el8.noarch 1/1 2021-09-02T07:26:02.778 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:02.778 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2021-09-02T07:26:02.779 INFO:teuthology.orchestra.run.smithi175.stdout: cephadm-2:17.0.0-7401.g9366f375.el8.noarch 2021-09-02T07:26:02.779 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:02.779 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:26:02.916 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-immutable-object-cache 2021-09-02T07:26:03.287 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:26:03.289 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:26:03.289 INFO:teuthology.orchestra.run.smithi197.stdout: Package Arch Version Repository Size 2021-09-02T07:26:03.289 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:26:03.290 INFO:teuthology.orchestra.run.smithi197.stdout:Installing: 2021-09-02T07:26:03.290 INFO:teuthology.orchestra.run.smithi197.stdout: cephadm noarch 2:17.0.0-7401.g9366f375.el8 ceph-noarch 69 k 2021-09-02T07:26:03.290 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:03.291 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction Summary 2021-09-02T07:26:03.291 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:26:03.291 INFO:teuthology.orchestra.run.smithi197.stdout:Install 1 Package 2021-09-02T07:26:03.291 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:03.292 INFO:teuthology.orchestra.run.smithi197.stdout:Total download size: 69 k 2021-09-02T07:26:03.292 INFO:teuthology.orchestra.run.smithi197.stdout:Installed size: 309 k 2021-09-02T07:26:03.292 INFO:teuthology.orchestra.run.smithi197.stdout:Downloading Packages: 2021-09-02T07:26:03.376 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:10 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:03.421 INFO:teuthology.orchestra.run.smithi197.stdout:cephadm-17.0.0-7401.g9366f375.el8.noarch.rpm 525 kB/s | 69 kB 00:00 2021-09-02T07:26:03.422 INFO:teuthology.orchestra.run.smithi197.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:26:03.423 INFO:teuthology.orchestra.run.smithi197.stdout:Total 519 kB/s | 69 kB 00:00 2021-09-02T07:26:03.424 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction check 2021-09-02T07:26:03.429 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction check succeeded. 2021-09-02T07:26:03.430 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction test 2021-09-02T07:26:03.431 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction test succeeded. 2021-09-02T07:26:03.434 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction 2021-09-02T07:26:03.487 INFO:teuthology.orchestra.run.smithi197.stdout: Preparing : 1/1 2021-09-02T07:26:04.326 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:26:04.326 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:26:04.327 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repo Size 2021-09-02T07:26:04.327 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:26:04.327 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2021-09-02T07:26:04.327 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-7401.g9366f375.el8 ceph 169 k 2021-09-02T07:26:04.328 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:04.328 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2021-09-02T07:26:04.328 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:26:04.328 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2021-09-02T07:26:04.329 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:04.330 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 169 k 2021-09-02T07:26:04.330 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 567 k 2021-09-02T07:26:04.330 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2021-09-02T07:26:04.863 INFO:teuthology.orchestra.run.smithi175.stdout:ceph-immutable-object-cache-17.0.0-7401.g9366f3 314 kB/s | 169 kB 00:00 2021-09-02T07:26:04.864 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:26:04.865 INFO:teuthology.orchestra.run.smithi175.stdout:Total 313 kB/s | 169 kB 00:00 2021-09-02T07:26:04.865 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2021-09-02T07:26:04.877 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2021-09-02T07:26:04.877 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2021-09-02T07:26:04.925 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2021-09-02T07:26:04.928 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2021-09-02T07:26:05.052 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2021-09-02T07:26:05.243 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-7401.g9366f375. 1/1 2021-09-02T07:26:05.620 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-7401.g9366f375. 1/1 2021-09-02T07:26:05.734 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: cephadm-2:17.0.0-7401.g9366f375.el8.noarch 1/1 2021-09-02T07:26:05.919 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-7401.g9366f375. 1/1 2021-09-02T07:26:05.919 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:05.920 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2021-09-02T07:26:05.920 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-immutable-object-cache-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:26:05.920 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:05.920 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:26:05.979 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : cephadm-2:17.0.0-7401.g9366f375.el8.noarch 1/1 2021-09-02T07:26:06.079 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr 2021-09-02T07:26:06.160 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: cephadm-2:17.0.0-7401.g9366f375.el8.noarch 1/1 2021-09-02T07:26:06.409 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : cephadm-2:17.0.0-7401.g9366f375.el8.noarch 1/1 2021-09-02T07:26:06.409 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:06.410 INFO:teuthology.orchestra.run.smithi197.stdout:Installed: 2021-09-02T07:26:06.410 INFO:teuthology.orchestra.run.smithi197.stdout: cephadm-2:17.0.0-7401.g9366f375.el8.noarch 2021-09-02T07:26:06.410 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:06.410 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:26:06.529 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:13 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:06.584 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install ceph-immutable-object-cache 2021-09-02T07:26:07.042 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:01:14 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:07.226 INFO:teuthology.orchestra.run.smithi175.stdout:Package ceph-mgr-2:17.0.0-7401.g9366f375.el8.x86_64 is already installed. 2021-09-02T07:26:07.474 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:26:07.475 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2021-09-02T07:26:07.475 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:26:07.531 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr-dashboard 2021-09-02T07:26:07.977 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:14 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:08.029 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:26:08.030 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:26:08.030 INFO:teuthology.orchestra.run.smithi197.stdout: Package Arch Version Repo Size 2021-09-02T07:26:08.030 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:26:08.030 INFO:teuthology.orchestra.run.smithi197.stdout:Installing: 2021-09-02T07:26:08.031 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-7401.g9366f375.el8 ceph 169 k 2021-09-02T07:26:08.031 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:08.031 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction Summary 2021-09-02T07:26:08.031 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:26:08.031 INFO:teuthology.orchestra.run.smithi197.stdout:Install 1 Package 2021-09-02T07:26:08.032 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:08.032 INFO:teuthology.orchestra.run.smithi197.stdout:Total download size: 169 k 2021-09-02T07:26:08.032 INFO:teuthology.orchestra.run.smithi197.stdout:Installed size: 567 k 2021-09-02T07:26:08.032 INFO:teuthology.orchestra.run.smithi197.stdout:Downloading Packages: 2021-09-02T07:26:08.205 INFO:teuthology.orchestra.run.smithi197.stdout:ceph-immutable-object-cache-17.0.0-7401.g9366f3 963 kB/s | 169 kB 00:00 2021-09-02T07:26:08.206 INFO:teuthology.orchestra.run.smithi197.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:26:08.207 INFO:teuthology.orchestra.run.smithi197.stdout:Total 956 kB/s | 169 kB 00:00 2021-09-02T07:26:08.207 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction check 2021-09-02T07:26:08.218 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction check succeeded. 2021-09-02T07:26:08.219 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction test 2021-09-02T07:26:08.266 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction test succeeded. 2021-09-02T07:26:08.269 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction 2021-09-02T07:26:08.396 INFO:teuthology.orchestra.run.smithi197.stdout: Preparing : 1/1 2021-09-02T07:26:08.555 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-7401.g9366f375. 1/1 2021-09-02T07:26:08.920 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-7401.g9366f375. 1/1 2021-09-02T07:26:09.621 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:26:09.622 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:26:09.622 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2021-09-02T07:26:09.622 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:26:09.623 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2021-09-02T07:26:09.623 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-dashboard noarch 2:17.0.0-7401.g9366f375.el8 ceph-noarch 1.5 M 2021-09-02T07:26:09.623 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2021-09-02T07:26:09.623 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-grafana-dashboards noarch 2:17.0.0-7401.g9366f375.el8 ceph-noarch 24 k 2021-09-02T07:26:09.624 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-prometheus-alerts noarch 2:17.0.0-7401.g9366f375.el8 ceph-noarch 8.9 k 2021-09-02T07:26:09.624 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-09-02T07:26:09.624 INFO:teuthology.orchestra.run.smithi175.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-09-02T07:26:09.624 INFO:teuthology.orchestra.run.smithi175.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-09-02T07:26:09.624 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:09.625 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2021-09-02T07:26:09.625 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:26:09.625 INFO:teuthology.orchestra.run.smithi175.stdout:Install 6 Packages 2021-09-02T07:26:09.625 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:09.647 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 1.8 M 2021-09-02T07:26:09.647 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 7.4 M 2021-09-02T07:26:09.647 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2021-09-02T07:26:09.757 INFO:teuthology.orchestra.run.smithi175.stdout:(1/6): ceph-prometheus-alerts-17.0.0-7401.g9366 82 kB/s | 8.9 kB 00:00 2021-09-02T07:26:09.765 INFO:teuthology.orchestra.run.smithi175.stdout:(2/6): ceph-grafana-dashboards-17.0.0-7401.g936 206 kB/s | 24 kB 00:00 2021-09-02T07:26:09.788 INFO:teuthology.orchestra.run.smithi175.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2021-09-02T07:26:09.839 INFO:teuthology.orchestra.run.smithi175.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 450 kB/s | 34 kB 00:00 2021-09-02T07:26:09.909 INFO:teuthology.orchestra.run.smithi175.stdout:(5/6): ceph-mgr-dashboard-17.0.0-7401.g9366f375 5.9 MB/s | 1.5 MB 00:00 2021-09-02T07:26:09.912 INFO:teuthology.orchestra.run.smithi175.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.6 MB/s | 196 kB 00:00 2021-09-02T07:26:09.915 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:26:09.915 INFO:teuthology.orchestra.run.smithi175.stdout:Total 6.8 MB/s | 1.8 MB 00:00 2021-09-02T07:26:09.916 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2021-09-02T07:26:09.923 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2021-09-02T07:26:09.925 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2021-09-02T07:26:10.008 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2021-09-02T07:26:10.011 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2021-09-02T07:26:10.165 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-7401.g9366f375. 1/1 2021-09-02T07:26:10.165 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:10.165 INFO:teuthology.orchestra.run.smithi197.stdout:Installed: 2021-09-02T07:26:10.166 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-immutable-object-cache-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:26:10.166 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:10.166 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:26:10.260 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2021-09-02T07:26:10.458 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-09-02T07:26:10.655 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-09-02T07:26:10.895 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-09-02T07:26:11.009 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install ceph-mgr 2021-09-02T07:26:11.049 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-7401.g9366f375.el8.n 4/6 2021-09-02T07:26:11.406 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-7401.g9366f375.el8. 5/6 2021-09-02T07:26:11.470 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:01:18 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:11.635 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-7401.g9366f375.el8.noarc 6/6 2021-09-02T07:26:11.851 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-7401.g9366f375.el8.noarc 6/6 2021-09-02T07:26:12.423 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-7401.g9366f375.el8. 1/6 2021-09-02T07:26:12.424 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-7401.g9366f375.el8.noarc 2/6 2021-09-02T07:26:12.424 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-7401.g9366f375.el8.n 3/6 2021-09-02T07:26:12.424 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-09-02T07:26:12.425 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-09-02T07:26:12.426 INFO:teuthology.orchestra.run.smithi197.stdout:Package ceph-mgr-2:17.0.0-7401.g9366f375.el8.x86_64 is already installed. 2021-09-02T07:26:12.427 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:26:12.427 INFO:teuthology.orchestra.run.smithi197.stdout:Nothing to do. 2021-09-02T07:26:12.427 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:26:12.428 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-09-02T07:26:12.428 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:12.428 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2021-09-02T07:26:12.428 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-grafana-dashboards-2:17.0.0-7401.g9366f375.el8.noarch 2021-09-02T07:26:12.429 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-dashboard-2:17.0.0-7401.g9366f375.el8.noarch 2021-09-02T07:26:12.429 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-prometheus-alerts-2:17.0.0-7401.g9366f375.el8.noarch 2021-09-02T07:26:12.429 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-09-02T07:26:12.429 INFO:teuthology.orchestra.run.smithi175.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-09-02T07:26:12.430 INFO:teuthology.orchestra.run.smithi175.stdout: python3-routes-2.4.1-12.el8.noarch 2021-09-02T07:26:12.430 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:12.430 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:26:12.431 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr-diskprediction-local 2021-09-02T07:26:12.483 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install ceph-mgr-dashboard 2021-09-02T07:26:12.875 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:19 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:12.927 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:01:19 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:13.851 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:26:13.852 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:26:13.853 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:26:13.854 INFO:teuthology.orchestra.run.smithi197.stdout: Package Arch Version Repository Size 2021-09-02T07:26:13.854 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:26:13.854 INFO:teuthology.orchestra.run.smithi197.stdout:Installing: 2021-09-02T07:26:13.854 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mgr-dashboard noarch 2:17.0.0-7401.g9366f375.el8 ceph-noarch 1.5 M 2021-09-02T07:26:13.855 INFO:teuthology.orchestra.run.smithi197.stdout:Installing dependencies: 2021-09-02T07:26:13.855 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-grafana-dashboards noarch 2:17.0.0-7401.g9366f375.el8 ceph-noarch 24 k 2021-09-02T07:26:13.855 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-prometheus-alerts noarch 2:17.0.0-7401.g9366f375.el8 ceph-noarch 8.9 k 2021-09-02T07:26:13.855 INFO:teuthology.orchestra.run.smithi197.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-09-02T07:26:13.856 INFO:teuthology.orchestra.run.smithi197.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-09-02T07:26:13.856 INFO:teuthology.orchestra.run.smithi197.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-09-02T07:26:13.856 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:13.856 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction Summary 2021-09-02T07:26:13.857 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:26:13.857 INFO:teuthology.orchestra.run.smithi197.stdout:Install 6 Packages 2021-09-02T07:26:13.857 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:13.857 INFO:teuthology.orchestra.run.smithi197.stdout:Total download size: 1.8 M 2021-09-02T07:26:13.858 INFO:teuthology.orchestra.run.smithi197.stdout:Installed size: 7.4 M 2021-09-02T07:26:13.858 INFO:teuthology.orchestra.run.smithi197.stdout:Downloading Packages: 2021-09-02T07:26:13.859 INFO:teuthology.orchestra.run.smithi175.stdout:===================================================================================================== 2021-09-02T07:26:13.859 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2021-09-02T07:26:13.860 INFO:teuthology.orchestra.run.smithi175.stdout:===================================================================================================== 2021-09-02T07:26:13.860 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2021-09-02T07:26:13.860 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-7401.g9366f375.el8 ceph-noarch 7.4 M 2021-09-02T07:26:13.860 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2021-09-02T07:26:13.861 INFO:teuthology.orchestra.run.smithi175.stdout: libgfortran x86_64 8.4.1-1.el8 CentOS-Base 639 k 2021-09-02T07:26:13.861 INFO:teuthology.orchestra.run.smithi175.stdout: libquadmath x86_64 8.4.1-1.el8 CentOS-Base 169 k 2021-09-02T07:26:13.861 INFO:teuthology.orchestra.run.smithi175.stdout: openblas x86_64 0.3.12-1.el8 CentOS-AppStream 4.6 M 2021-09-02T07:26:13.862 INFO:teuthology.orchestra.run.smithi175.stdout: openblas-threads x86_64 0.3.12-1.el8 CentOS-AppStream 4.7 M 2021-09-02T07:26:13.862 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-09-02T07:26:13.862 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-09-02T07:26:13.862 INFO:teuthology.orchestra.run.smithi175.stdout: python3-scipy x86_64 1.0.0-20.module_el8.4.0+790+083e3d81 CentOS-AppStream 15 M 2021-09-02T07:26:13.863 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:13.863 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2021-09-02T07:26:13.863 INFO:teuthology.orchestra.run.smithi175.stdout:===================================================================================================== 2021-09-02T07:26:13.863 INFO:teuthology.orchestra.run.smithi175.stdout:Install 8 Packages 2021-09-02T07:26:13.864 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:13.864 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 36 M 2021-09-02T07:26:13.864 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 197 M 2021-09-02T07:26:13.864 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2021-09-02T07:26:13.953 INFO:teuthology.orchestra.run.smithi175.stdout:(1/8): openblas-0.3.12-1.el8.x86_64.rpm 47 MB/s | 4.6 MB 00:00 2021-09-02T07:26:14.004 INFO:teuthology.orchestra.run.smithi197.stdout:(1/6): ceph-prometheus-alerts-17.0.0-7401.g9366 59 kB/s | 8.9 kB 00:00 2021-09-02T07:26:14.008 INFO:teuthology.orchestra.run.smithi175.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 68 MB/s | 3.7 MB 00:00 2021-09-02T07:26:14.015 INFO:teuthology.orchestra.run.smithi175.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 42 MB/s | 225 kB 00:00 2021-09-02T07:26:14.016 INFO:teuthology.orchestra.run.smithi197.stdout:(2/6): ceph-grafana-dashboards-17.0.0-7401.g936 147 kB/s | 24 kB 00:00 2021-09-02T07:26:14.057 INFO:teuthology.orchestra.run.smithi197.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 798 kB/s | 43 kB 00:00 2021-09-02T07:26:14.079 INFO:teuthology.orchestra.run.smithi175.stdout:(4/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2021-09-02T07:26:14.090 INFO:teuthology.orchestra.run.smithi197.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 455 kB/s | 34 kB 00:00 2021-09-02T07:26:14.106 INFO:teuthology.orchestra.run.smithi175.stdout:(5/8): libgfortran-8.4.1-1.el8.x86_64.rpm 23 MB/s | 639 kB 00:00 2021-09-02T07:26:14.114 INFO:teuthology.orchestra.run.smithi175.stdout:(6/8): libquadmath-8.4.1-1.el8.x86_64.rpm 20 MB/s | 169 kB 00:00 2021-09-02T07:26:14.159 INFO:teuthology.orchestra.run.smithi197.stdout:(5/6): ceph-mgr-dashboard-17.0.0-7401.g9366f375 5.0 MB/s | 1.5 MB 00:00 2021-09-02T07:26:14.199 INFO:teuthology.orchestra.run.smithi197.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.4 MB/s | 196 kB 00:00 2021-09-02T07:26:14.202 INFO:teuthology.orchestra.run.smithi197.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:26:14.202 INFO:teuthology.orchestra.run.smithi197.stdout:Total 5.2 MB/s | 1.8 MB 00:00 2021-09-02T07:26:14.203 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction check 2021-09-02T07:26:14.211 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction check succeeded. 2021-09-02T07:26:14.212 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction test 2021-09-02T07:26:14.272 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction test succeeded. 2021-09-02T07:26:14.273 INFO:teuthology.orchestra.run.smithi175.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.4.0+79 56 MB/s | 15 MB 00:00 2021-09-02T07:26:14.276 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction 2021-09-02T07:26:14.384 INFO:teuthology.orchestra.run.smithi175.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-740 14 MB/s | 7.4 MB 00:00 2021-09-02T07:26:14.387 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:26:14.388 INFO:teuthology.orchestra.run.smithi175.stdout:Total 67 MB/s | 36 MB 00:00 2021-09-02T07:26:14.397 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2021-09-02T07:26:14.415 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2021-09-02T07:26:14.416 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2021-09-02T07:26:14.547 INFO:teuthology.orchestra.run.smithi197.stdout: Preparing : 1/1 2021-09-02T07:26:14.745 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-09-02T07:26:14.776 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2021-09-02T07:26:14.779 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2021-09-02T07:26:14.964 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-09-02T07:26:15.160 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-09-02T07:26:15.311 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-7401.g9366f375.el8.n 4/6 2021-09-02T07:26:15.331 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2021-09-02T07:26:15.516 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2021-09-02T07:26:15.632 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2021-09-02T07:26:15.670 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-7401.g9366f375.el8. 5/6 2021-09-02T07:26:15.809 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2021-09-02T07:26:15.898 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-7401.g9366f375.el8.noarc 6/6 2021-09-02T07:26:16.127 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-7401.g9366f375.el8.noarc 6/6 2021-09-02T07:26:16.128 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-7401.g9366f375.el8. 1/6 2021-09-02T07:26:16.128 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-7401.g9366f375.el8.noarc 2/6 2021-09-02T07:26:16.129 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-7401.g9366f375.el8.n 3/6 2021-09-02T07:26:16.129 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-09-02T07:26:16.129 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-09-02T07:26:16.318 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2021-09-02T07:26:16.511 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : openblas-0.3.12-1.el8.x86_64 3/8 2021-09-02T07:26:16.721 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-09-02T07:26:16.722 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:16.726 INFO:teuthology.orchestra.run.smithi197.stdout:Installed: 2021-09-02T07:26:16.726 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-grafana-dashboards-2:17.0.0-7401.g9366f375.el8.noarch 2021-09-02T07:26:16.728 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mgr-dashboard-2:17.0.0-7401.g9366f375.el8.noarch 2021-09-02T07:26:16.729 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-prometheus-alerts-2:17.0.0-7401.g9366f375.el8.noarch 2021-09-02T07:26:16.729 INFO:teuthology.orchestra.run.smithi197.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-09-02T07:26:16.729 INFO:teuthology.orchestra.run.smithi197.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-09-02T07:26:16.729 INFO:teuthology.orchestra.run.smithi197.stdout: python3-routes-2.4.1-12.el8.noarch 2021-09-02T07:26:16.730 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:16.730 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:26:16.823 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install ceph-mgr-diskprediction-local 2021-09-02T07:26:17.042 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 3/8 2021-09-02T07:26:17.224 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-09-02T07:26:17.281 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:01:24 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:17.942 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-09-02T07:26:18.234 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:26:18.236 INFO:teuthology.orchestra.run.smithi197.stdout:===================================================================================================== 2021-09-02T07:26:18.236 INFO:teuthology.orchestra.run.smithi197.stdout: Package Arch Version Repository Size 2021-09-02T07:26:18.237 INFO:teuthology.orchestra.run.smithi197.stdout:===================================================================================================== 2021-09-02T07:26:18.237 INFO:teuthology.orchestra.run.smithi197.stdout:Installing: 2021-09-02T07:26:18.237 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-7401.g9366f375.el8 ceph-noarch 7.4 M 2021-09-02T07:26:18.237 INFO:teuthology.orchestra.run.smithi197.stdout:Installing dependencies: 2021-09-02T07:26:18.237 INFO:teuthology.orchestra.run.smithi197.stdout: libgfortran x86_64 8.4.1-1.el8 CentOS-Base 639 k 2021-09-02T07:26:18.238 INFO:teuthology.orchestra.run.smithi197.stdout: libquadmath x86_64 8.4.1-1.el8 CentOS-Base 169 k 2021-09-02T07:26:18.238 INFO:teuthology.orchestra.run.smithi197.stdout: openblas x86_64 0.3.12-1.el8 CentOS-AppStream 4.6 M 2021-09-02T07:26:18.238 INFO:teuthology.orchestra.run.smithi197.stdout: openblas-threads x86_64 0.3.12-1.el8 CentOS-AppStream 4.7 M 2021-09-02T07:26:18.239 INFO:teuthology.orchestra.run.smithi197.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-09-02T07:26:18.240 INFO:teuthology.orchestra.run.smithi197.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-09-02T07:26:18.240 INFO:teuthology.orchestra.run.smithi197.stdout: python3-scipy x86_64 1.0.0-20.module_el8.4.0+790+083e3d81 CentOS-AppStream 15 M 2021-09-02T07:26:18.241 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:18.241 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction Summary 2021-09-02T07:26:18.241 INFO:teuthology.orchestra.run.smithi197.stdout:===================================================================================================== 2021-09-02T07:26:18.241 INFO:teuthology.orchestra.run.smithi197.stdout:Install 8 Packages 2021-09-02T07:26:18.241 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:18.242 INFO:teuthology.orchestra.run.smithi197.stdout:Total download size: 36 M 2021-09-02T07:26:18.242 INFO:teuthology.orchestra.run.smithi197.stdout:Installed size: 197 M 2021-09-02T07:26:18.242 INFO:teuthology.orchestra.run.smithi197.stdout:Downloading Packages: 2021-09-02T07:26:18.283 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-09-02T07:26:18.347 INFO:teuthology.orchestra.run.smithi197.stdout:(1/8): openblas-0.3.12-1.el8.x86_64.rpm 44 MB/s | 4.6 MB 00:00 2021-09-02T07:26:18.403 INFO:teuthology.orchestra.run.smithi197.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 62 MB/s | 3.7 MB 00:00 2021-09-02T07:26:18.408 INFO:teuthology.orchestra.run.smithi197.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 41 MB/s | 225 kB 00:00 2021-09-02T07:26:18.428 INFO:teuthology.orchestra.run.smithi197.stdout:(4/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2021-09-02T07:26:18.449 INFO:teuthology.orchestra.run.smithi197.stdout:(5/8): libgfortran-8.4.1-1.el8.x86_64.rpm 29 MB/s | 639 kB 00:00 2021-09-02T07:26:18.456 INFO:teuthology.orchestra.run.smithi197.stdout:(6/8): libquadmath-8.4.1-1.el8.x86_64.rpm 25 MB/s | 169 kB 00:00 2021-09-02T07:26:18.482 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-02T07:26:18.675 INFO:teuthology.orchestra.run.smithi197.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.4.0+79 55 MB/s | 15 MB 00:00 2021-09-02T07:26:18.735 INFO:teuthology.orchestra.run.smithi197.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-740 15 MB/s | 7.4 MB 00:00 2021-09-02T07:26:18.738 INFO:teuthology.orchestra.run.smithi197.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:26:18.738 INFO:teuthology.orchestra.run.smithi197.stdout:Total 72 MB/s | 36 MB 00:00 2021-09-02T07:26:18.747 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction check 2021-09-02T07:26:18.766 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction check succeeded. 2021-09-02T07:26:18.766 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction test 2021-09-02T07:26:19.124 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction test succeeded. 2021-09-02T07:26:19.128 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction 2021-09-02T07:26:19.703 INFO:teuthology.orchestra.run.smithi197.stdout: Preparing : 1/1 2021-09-02T07:26:19.820 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2021-09-02T07:26:19.961 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2021-09-02T07:26:20.094 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2021-09-02T07:26:20.475 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-02T07:26:20.650 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2021-09-02T07:26:20.819 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : openblas-0.3.12-1.el8.x86_64 3/8 2021-09-02T07:26:21.349 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 3/8 2021-09-02T07:26:21.470 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-09-02T07:26:21.718 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-scipy-1.0.0-20.module_el8.4.0+790+083e3d81.x 7/8 2021-09-02T07:26:21.899 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-7401.g9366f37 8/8 2021-09-02T07:26:22.172 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-7401.g9366f37 8/8 2021-09-02T07:26:22.172 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-7401.g9366f37 1/8 2021-09-02T07:26:22.173 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 2/8 2021-09-02T07:26:22.173 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 3/8 2021-09-02T07:26:22.173 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-09-02T07:26:22.173 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-09-02T07:26:22.174 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.4.0+790+083e3d81.x 6/8 2021-09-02T07:26:22.174 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 7/8 2021-09-02T07:26:22.174 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-09-02T07:26:22.474 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 8/8 2021-09-02T07:26:22.475 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:22.475 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2021-09-02T07:26:22.475 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-diskprediction-local-2:17.0.0-7401.g9366f375.el8.noarch 2021-09-02T07:26:22.475 INFO:teuthology.orchestra.run.smithi175.stdout: libgfortran-8.4.1-1.el8.x86_64 2021-09-02T07:26:22.476 INFO:teuthology.orchestra.run.smithi175.stdout: libquadmath-8.4.1-1.el8.x86_64 2021-09-02T07:26:22.476 INFO:teuthology.orchestra.run.smithi175.stdout: openblas-0.3.12-1.el8.x86_64 2021-09-02T07:26:22.476 INFO:teuthology.orchestra.run.smithi175.stdout: openblas-threads-0.3.12-1.el8.x86_64 2021-09-02T07:26:22.476 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-09-02T07:26:22.476 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-09-02T07:26:22.477 INFO:teuthology.orchestra.run.smithi175.stdout: python3-scipy-1.0.0-20.module_el8.4.0+790+083e3d81.x86_64 2021-09-02T07:26:22.477 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:22.477 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:26:22.493 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-09-02T07:26:22.559 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr-rook 2021-09-02T07:26:22.727 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-02T07:26:23.001 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:30 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:23.945 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:26:23.947 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================== 2021-09-02T07:26:23.947 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2021-09-02T07:26:23.948 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================== 2021-09-02T07:26:23.948 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2021-09-02T07:26:23.948 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-rook noarch 2:17.0.0-7401.g9366f375.el8 ceph-noarch 44 k 2021-09-02T07:26:23.948 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2021-09-02T07:26:23.949 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-09-02T07:26:23.949 INFO:teuthology.orchestra.run.smithi175.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-09-02T07:26:23.949 INFO:teuthology.orchestra.run.smithi175.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-09-02T07:26:23.949 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-09-02T07:26:23.949 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-09-02T07:26:23.950 INFO:teuthology.orchestra.run.smithi175.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-09-02T07:26:23.950 INFO:teuthology.orchestra.run.smithi175.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-09-02T07:26:23.950 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-09-02T07:26:23.951 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-09-02T07:26:23.951 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-09-02T07:26:23.951 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-09-02T07:26:23.951 INFO:teuthology.orchestra.run.smithi175.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-09-02T07:26:23.951 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:23.952 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2021-09-02T07:26:23.952 INFO:teuthology.orchestra.run.smithi175.stdout:======================================================================================== 2021-09-02T07:26:23.952 INFO:teuthology.orchestra.run.smithi175.stdout:Install 13 Packages 2021-09-02T07:26:23.952 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:23.953 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 1.9 M 2021-09-02T07:26:23.953 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 25 M 2021-09-02T07:26:23.954 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2021-09-02T07:26:23.981 INFO:teuthology.orchestra.run.smithi175.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 877 kB/s | 27 kB 00:00 2021-09-02T07:26:23.984 INFO:teuthology.orchestra.run.smithi175.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 38 MB/s | 126 kB 00:00 2021-09-02T07:26:23.984 INFO:teuthology.orchestra.run.smithi175.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 584 kB/s | 20 kB 00:00 2021-09-02T07:26:23.986 INFO:teuthology.orchestra.run.smithi175.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 44 MB/s | 110 kB 00:00 2021-09-02T07:26:23.988 INFO:teuthology.orchestra.run.smithi175.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 27 MB/s | 43 kB 00:00 2021-09-02T07:26:23.989 INFO:teuthology.orchestra.run.smithi175.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 33 MB/s | 155 kB 00:00 2021-09-02T07:26:24.081 INFO:teuthology.orchestra.run.smithi175.stdout:(7/13): ceph-mgr-rook-17.0.0-7401.g9366f375.el8 336 kB/s | 44 kB 00:00 2021-09-02T07:26:24.081 INFO:teuthology.orchestra.run.smithi175.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 178 kB/s | 16 kB 00:00 2021-09-02T07:26:24.094 INFO:teuthology.orchestra.run.smithi175.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 313 kB/s | 33 kB 00:00 2021-09-02T07:26:24.158 INFO:teuthology.orchestra.run.smithi175.stdout:(10/13): python3-rsa-3.4.2-16.el8.noarch.rpm 1.1 MB/s | 73 kB 00:00 2021-09-02T07:26:24.169 INFO:teuthology.orchestra.run.smithi175.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 918 kB/s | 81 kB 00:00 2021-09-02T07:26:24.212 INFO:teuthology.orchestra.run.smithi175.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.1 MB/s | 61 kB 00:00 2021-09-02T07:26:24.303 INFO:teuthology.orchestra.run.smithi175.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.3 MB/s | 1.2 MB 00:00 2021-09-02T07:26:24.305 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:26:24.305 INFO:teuthology.orchestra.run.smithi175.stdout:Total 5.4 MB/s | 1.9 MB 00:00 2021-09-02T07:26:24.310 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2021-09-02T07:26:24.324 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2021-09-02T07:26:24.324 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2021-09-02T07:26:24.399 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2021-09-02T07:26:24.402 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2021-09-02T07:26:24.695 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-02T07:26:24.890 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2021-09-02T07:26:25.120 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-09-02T07:26:25.970 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-scipy-1.0.0-20.module_el8.4.0+790+083e3d81.x 7/8 2021-09-02T07:26:26.152 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-7401.g9366f37 8/8 2021-09-02T07:26:26.283 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-09-02T07:26:26.404 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-7401.g9366f37 8/8 2021-09-02T07:26:26.404 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-7401.g9366f37 1/8 2021-09-02T07:26:26.404 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 2/8 2021-09-02T07:26:26.404 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 3/8 2021-09-02T07:26:26.405 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-09-02T07:26:26.405 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-09-02T07:26:26.405 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.4.0+790+083e3d81.x 6/8 2021-09-02T07:26:26.405 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 7/8 2021-09-02T07:26:26.506 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 3/13 2021-09-02T07:26:26.724 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-09-02T07:26:26.734 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 8/8 2021-09-02T07:26:26.735 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:26.735 INFO:teuthology.orchestra.run.smithi197.stdout:Installed: 2021-09-02T07:26:26.735 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mgr-diskprediction-local-2:17.0.0-7401.g9366f375.el8.noarch 2021-09-02T07:26:26.735 INFO:teuthology.orchestra.run.smithi197.stdout: libgfortran-8.4.1-1.el8.x86_64 2021-09-02T07:26:26.736 INFO:teuthology.orchestra.run.smithi197.stdout: libquadmath-8.4.1-1.el8.x86_64 2021-09-02T07:26:26.736 INFO:teuthology.orchestra.run.smithi197.stdout: openblas-0.3.12-1.el8.x86_64 2021-09-02T07:26:26.736 INFO:teuthology.orchestra.run.smithi197.stdout: openblas-threads-0.3.12-1.el8.x86_64 2021-09-02T07:26:26.736 INFO:teuthology.orchestra.run.smithi197.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-09-02T07:26:26.737 INFO:teuthology.orchestra.run.smithi197.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-09-02T07:26:26.737 INFO:teuthology.orchestra.run.smithi197.stdout: python3-scipy-1.0.0-20.module_el8.4.0+790+083e3d81.x86_64 2021-09-02T07:26:26.737 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:26.737 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:26:26.812 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install ceph-mgr-rook 2021-09-02T07:26:26.917 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-09-02T07:26:27.163 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-09-02T07:26:27.258 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:01:34 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:27.440 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-09-02T07:26:27.614 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-09-02T07:26:28.223 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:26:28.225 INFO:teuthology.orchestra.run.smithi197.stdout:======================================================================================== 2021-09-02T07:26:28.225 INFO:teuthology.orchestra.run.smithi197.stdout: Package Arch Version Repository Size 2021-09-02T07:26:28.226 INFO:teuthology.orchestra.run.smithi197.stdout:======================================================================================== 2021-09-02T07:26:28.226 INFO:teuthology.orchestra.run.smithi197.stdout:Installing: 2021-09-02T07:26:28.226 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mgr-rook noarch 2:17.0.0-7401.g9366f375.el8 ceph-noarch 44 k 2021-09-02T07:26:28.226 INFO:teuthology.orchestra.run.smithi197.stdout:Installing dependencies: 2021-09-02T07:26:28.227 INFO:teuthology.orchestra.run.smithi197.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-09-02T07:26:28.227 INFO:teuthology.orchestra.run.smithi197.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-09-02T07:26:28.227 INFO:teuthology.orchestra.run.smithi197.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-09-02T07:26:28.227 INFO:teuthology.orchestra.run.smithi197.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-09-02T07:26:28.227 INFO:teuthology.orchestra.run.smithi197.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-09-02T07:26:28.228 INFO:teuthology.orchestra.run.smithi197.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-09-02T07:26:28.228 INFO:teuthology.orchestra.run.smithi197.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-09-02T07:26:28.228 INFO:teuthology.orchestra.run.smithi197.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-09-02T07:26:28.228 INFO:teuthology.orchestra.run.smithi197.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-09-02T07:26:28.229 INFO:teuthology.orchestra.run.smithi197.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-09-02T07:26:28.229 INFO:teuthology.orchestra.run.smithi197.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-09-02T07:26:28.229 INFO:teuthology.orchestra.run.smithi197.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-09-02T07:26:28.229 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:28.229 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction Summary 2021-09-02T07:26:28.230 INFO:teuthology.orchestra.run.smithi197.stdout:======================================================================================== 2021-09-02T07:26:28.230 INFO:teuthology.orchestra.run.smithi197.stdout:Install 13 Packages 2021-09-02T07:26:28.230 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:28.231 INFO:teuthology.orchestra.run.smithi197.stdout:Total download size: 1.9 M 2021-09-02T07:26:28.231 INFO:teuthology.orchestra.run.smithi197.stdout:Installed size: 25 M 2021-09-02T07:26:28.232 INFO:teuthology.orchestra.run.smithi197.stdout:Downloading Packages: 2021-09-02T07:26:28.259 INFO:teuthology.orchestra.run.smithi197.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 877 kB/s | 27 kB 00:00 2021-09-02T07:26:28.263 INFO:teuthology.orchestra.run.smithi197.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 35 MB/s | 126 kB 00:00 2021-09-02T07:26:28.263 INFO:teuthology.orchestra.run.smithi197.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 578 kB/s | 20 kB 00:00 2021-09-02T07:26:28.265 INFO:teuthology.orchestra.run.smithi197.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 34 MB/s | 110 kB 00:00 2021-09-02T07:26:28.271 INFO:teuthology.orchestra.run.smithi197.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 19 MB/s | 155 kB 00:00 2021-09-02T07:26:28.271 INFO:teuthology.orchestra.run.smithi197.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 7.1 MB/s | 43 kB 00:00 2021-09-02T07:26:28.345 INFO:teuthology.orchestra.run.smithi197.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 223 kB/s | 16 kB 00:00 2021-09-02T07:26:28.353 INFO:teuthology.orchestra.run.smithi197.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 401 kB/s | 33 kB 00:00 2021-09-02T07:26:28.418 INFO:teuthology.orchestra.run.smithi197.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 1.1 MB/s | 81 kB 00:00 2021-09-02T07:26:28.443 INFO:teuthology.orchestra.run.smithi197.stdout:(10/13): python3-rsa-3.4.2-16.el8.noarch.rpm 2.8 MB/s | 73 kB 00:00 2021-09-02T07:26:28.469 INFO:teuthology.orchestra.run.smithi197.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 2.4 MB/s | 61 kB 00:00 2021-09-02T07:26:28.492 INFO:teuthology.orchestra.run.smithi197.stdout:(12/13): ceph-mgr-rook-17.0.0-7401.g9366f375.el 167 kB/s | 44 kB 00:00 2021-09-02T07:26:28.543 INFO:teuthology.orchestra.run.smithi197.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 6.2 MB/s | 1.2 MB 00:00 2021-09-02T07:26:28.545 INFO:teuthology.orchestra.run.smithi197.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:26:28.546 INFO:teuthology.orchestra.run.smithi197.stdout:Total 6.1 MB/s | 1.9 MB 00:00 2021-09-02T07:26:28.550 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction check 2021-09-02T07:26:28.564 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction check succeeded. 2021-09-02T07:26:28.564 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction test 2021-09-02T07:26:28.638 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction test succeeded. 2021-09-02T07:26:28.641 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction 2021-09-02T07:26:28.763 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-09-02T07:26:29.013 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-09-02T07:26:29.143 INFO:teuthology.orchestra.run.smithi197.stdout: Preparing : 1/1 2021-09-02T07:26:29.179 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-09-02T07:26:29.340 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-09-02T07:26:29.389 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-09-02T07:26:29.530 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-rook-2:17.0.0-7401.g9366f375.el8.noarch 13/13 2021-09-02T07:26:29.583 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-09-02T07:26:29.774 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-7401.g9366f375.el8.noarch 13/13 2021-09-02T07:26:29.774 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-rook-2:17.0.0-7401.g9366f375.el8.noarch 1/13 2021-09-02T07:26:29.775 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-09-02T07:26:29.775 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-09-02T07:26:29.775 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-09-02T07:26:29.775 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-09-02T07:26:29.776 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-09-02T07:26:29.776 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-09-02T07:26:29.776 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-09-02T07:26:29.776 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-09-02T07:26:29.777 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-09-02T07:26:29.777 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-09-02T07:26:29.777 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 12/13 2021-09-02T07:26:29.813 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 3/13 2021-09-02T07:26:30.067 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-09-02T07:26:30.123 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-09-02T07:26:30.123 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:30.124 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2021-09-02T07:26:30.124 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-rook-2:17.0.0-7401.g9366f375.el8.noarch 2021-09-02T07:26:30.124 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-09-02T07:26:30.124 INFO:teuthology.orchestra.run.smithi175.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-09-02T07:26:30.125 INFO:teuthology.orchestra.run.smithi175.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-09-02T07:26:30.125 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-09-02T07:26:30.125 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-09-02T07:26:30.125 INFO:teuthology.orchestra.run.smithi175.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-09-02T07:26:30.125 INFO:teuthology.orchestra.run.smithi175.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-09-02T07:26:30.126 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-09-02T07:26:30.126 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-09-02T07:26:30.126 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-09-02T07:26:30.127 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-09-02T07:26:30.127 INFO:teuthology.orchestra.run.smithi175.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-09-02T07:26:30.127 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:30.127 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:26:30.192 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr-cephadm 2021-09-02T07:26:30.259 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-09-02T07:26:30.477 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-09-02T07:26:30.636 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:37 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:30.739 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-09-02T07:26:30.958 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-09-02T07:26:31.614 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:26:31.616 INFO:teuthology.orchestra.run.smithi175.stdout:=================================================================================================================== 2021-09-02T07:26:31.616 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2021-09-02T07:26:31.616 INFO:teuthology.orchestra.run.smithi175.stdout:=================================================================================================================== 2021-09-02T07:26:31.617 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2021-09-02T07:26:31.617 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-cephadm noarch 2:17.0.0-7401.g9366f375.el8 ceph-noarch 91 k 2021-09-02T07:26:31.617 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2021-09-02T07:26:31.617 INFO:teuthology.orchestra.run.smithi175.stdout: python3-asyncssh noarch 2.7.0-1.el8 copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 476 k 2021-09-02T07:26:31.617 INFO:teuthology.orchestra.run.smithi175.stdout: python3-babel noarch 2.5.1-5.el8 CentOS-AppStream 4.8 M 2021-09-02T07:26:31.618 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 CentOS-AppStream 538 k 2021-09-02T07:26:31.618 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:31.618 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2021-09-02T07:26:31.618 INFO:teuthology.orchestra.run.smithi175.stdout:=================================================================================================================== 2021-09-02T07:26:31.619 INFO:teuthology.orchestra.run.smithi175.stdout:Install 4 Packages 2021-09-02T07:26:31.621 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:31.621 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 5.9 M 2021-09-02T07:26:31.621 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 25 M 2021-09-02T07:26:31.621 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2021-09-02T07:26:31.668 INFO:teuthology.orchestra.run.smithi175.stdout:(1/4): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 10 MB/s | 538 kB 00:00 2021-09-02T07:26:31.710 INFO:teuthology.orchestra.run.smithi175.stdout:(2/4): python3-babel-2.5.1-5.el8.noarch.rpm 51 MB/s | 4.8 MB 00:00 2021-09-02T07:26:31.764 INFO:teuthology.orchestra.run.smithi175.stdout:(3/4): ceph-mgr-cephadm-17.0.0-7401.g9366f375.e 619 kB/s | 91 kB 00:00 2021-09-02T07:26:32.054 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-09-02T07:26:32.342 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-09-02T07:26:32.525 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-09-02T07:26:32.972 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-09-02T07:26:33.151 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : ceph-mgr-rook-2:17.0.0-7401.g9366f375.el8.noarch 13/13 2021-09-02T07:26:33.402 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-7401.g9366f375.el8.noarch 13/13 2021-09-02T07:26:33.403 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : ceph-mgr-rook-2:17.0.0-7401.g9366f375.el8.noarch 1/13 2021-09-02T07:26:33.403 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-09-02T07:26:33.403 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-09-02T07:26:33.404 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-09-02T07:26:33.404 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-09-02T07:26:33.404 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-09-02T07:26:33.404 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-09-02T07:26:33.405 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-09-02T07:26:33.405 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-09-02T07:26:33.405 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-09-02T07:26:33.405 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-09-02T07:26:33.405 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 12/13 2021-09-02T07:26:33.514 INFO:teuthology.orchestra.run.smithi175.stdout:(4/4): python3-asyncssh-2.7.0-1.el8.noarch.rpm 258 kB/s | 476 kB 00:01 2021-09-02T07:26:33.520 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:26:33.520 INFO:teuthology.orchestra.run.smithi175.stdout:Total 3.1 MB/s | 5.9 MB 00:01 2021-09-02T07:26:33.523 INFO:teuthology.orchestra.run.smithi175.stderr:warning: /var/cache/dnf/copr:copr.fedorainfracloud.org:ceph:python3-asyncssh-5dfbfa110d1e0dad/packages/python3-asyncssh-2.7.0-1.el8.noarch.rpm: Header V3 RSA/SHA1 Signature, key ID 45bf1189: NOKEY 2021-09-02T07:26:33.742 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-09-02T07:26:33.743 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:33.743 INFO:teuthology.orchestra.run.smithi197.stdout:Installed: 2021-09-02T07:26:33.744 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mgr-rook-2:17.0.0-7401.g9366f375.el8.noarch 2021-09-02T07:26:33.744 INFO:teuthology.orchestra.run.smithi197.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-09-02T07:26:33.745 INFO:teuthology.orchestra.run.smithi197.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-09-02T07:26:33.745 INFO:teuthology.orchestra.run.smithi197.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-09-02T07:26:33.746 INFO:teuthology.orchestra.run.smithi197.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-09-02T07:26:33.746 INFO:teuthology.orchestra.run.smithi197.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-09-02T07:26:33.746 INFO:teuthology.orchestra.run.smithi197.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-09-02T07:26:33.746 INFO:teuthology.orchestra.run.smithi197.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-09-02T07:26:33.747 INFO:teuthology.orchestra.run.smithi197.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-09-02T07:26:33.747 INFO:teuthology.orchestra.run.smithi197.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-09-02T07:26:33.747 INFO:teuthology.orchestra.run.smithi197.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-09-02T07:26:33.747 INFO:teuthology.orchestra.run.smithi197.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-09-02T07:26:33.748 INFO:teuthology.orchestra.run.smithi197.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-09-02T07:26:33.748 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:33.749 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:26:33.830 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install ceph-mgr-cephadm 2021-09-02T07:26:34.275 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:01:41 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:34.333 INFO:teuthology.orchestra.run.smithi175.stdout:Copr repo for python3-asyncssh owned by ceph 1.2 kB/s | 1.0 kB 00:00 2021-09-02T07:26:34.448 INFO:teuthology.orchestra.run.smithi175.stderr:Importing GPG key 0x45BF1189: 2021-09-02T07:26:34.448 INFO:teuthology.orchestra.run.smithi175.stderr: Userid : "ceph_python3-asyncssh (None) " 2021-09-02T07:26:34.449 INFO:teuthology.orchestra.run.smithi175.stderr: Fingerprint: FD53 FC11 9FA9 2C47 E35D 9C35 339B 8950 45BF 1189 2021-09-02T07:26:34.449 INFO:teuthology.orchestra.run.smithi175.stderr: From : https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/pubkey.gpg 2021-09-02T07:26:34.546 INFO:teuthology.orchestra.run.smithi175.stdout:Key imported successfully 2021-09-02T07:26:34.547 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2021-09-02T07:26:34.557 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2021-09-02T07:26:34.557 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2021-09-02T07:26:34.644 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2021-09-02T07:26:34.647 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2021-09-02T07:26:34.977 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2021-09-02T07:26:35.207 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:26:35.208 INFO:teuthology.orchestra.run.smithi197.stdout:=================================================================================================================== 2021-09-02T07:26:35.208 INFO:teuthology.orchestra.run.smithi197.stdout: Package Arch Version Repository Size 2021-09-02T07:26:35.208 INFO:teuthology.orchestra.run.smithi197.stdout:=================================================================================================================== 2021-09-02T07:26:35.209 INFO:teuthology.orchestra.run.smithi197.stdout:Installing: 2021-09-02T07:26:35.209 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mgr-cephadm noarch 2:17.0.0-7401.g9366f375.el8 ceph-noarch 91 k 2021-09-02T07:26:35.210 INFO:teuthology.orchestra.run.smithi197.stdout:Installing dependencies: 2021-09-02T07:26:35.210 INFO:teuthology.orchestra.run.smithi197.stdout: python3-asyncssh noarch 2.7.0-1.el8 copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 476 k 2021-09-02T07:26:35.210 INFO:teuthology.orchestra.run.smithi197.stdout: python3-babel noarch 2.5.1-5.el8 CentOS-AppStream 4.8 M 2021-09-02T07:26:35.210 INFO:teuthology.orchestra.run.smithi197.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 CentOS-AppStream 538 k 2021-09-02T07:26:35.210 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:35.211 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction Summary 2021-09-02T07:26:35.211 INFO:teuthology.orchestra.run.smithi197.stdout:=================================================================================================================== 2021-09-02T07:26:35.211 INFO:teuthology.orchestra.run.smithi197.stdout:Install 4 Packages 2021-09-02T07:26:35.211 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:35.212 INFO:teuthology.orchestra.run.smithi197.stdout:Total download size: 5.9 M 2021-09-02T07:26:35.213 INFO:teuthology.orchestra.run.smithi197.stdout:Installed size: 25 M 2021-09-02T07:26:35.213 INFO:teuthology.orchestra.run.smithi197.stdout:Downloading Packages: 2021-09-02T07:26:35.270 INFO:teuthology.orchestra.run.smithi197.stdout:(1/4): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 8.8 MB/s | 538 kB 00:00 2021-09-02T07:26:35.302 INFO:teuthology.orchestra.run.smithi197.stdout:(2/4): python3-babel-2.5.1-5.el8.noarch.rpm 52 MB/s | 4.8 MB 00:00 2021-09-02T07:26:35.340 INFO:teuthology.orchestra.run.smithi197.stdout:(3/4): ceph-mgr-cephadm-17.0.0-7401.g9366f375.e 700 kB/s | 91 kB 00:00 2021-09-02T07:26:35.983 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-asyncssh-2.7.0-1.el8.noarch 1/4 2021-09-02T07:26:36.330 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 2/4 2021-09-02T07:26:36.825 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 3/4 2021-09-02T07:26:37.004 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-7401.g9366f375.el8.noarch 4/4 2021-09-02T07:26:37.214 INFO:teuthology.orchestra.run.smithi197.stdout:(4/4): python3-asyncssh-2.7.0-1.el8.noarch.rpm 245 kB/s | 476 kB 00:01 2021-09-02T07:26:37.219 INFO:teuthology.orchestra.run.smithi197.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:26:37.220 INFO:teuthology.orchestra.run.smithi197.stdout:Total 2.9 MB/s | 5.9 MB 00:02 2021-09-02T07:26:37.224 INFO:teuthology.orchestra.run.smithi197.stderr:warning: /var/cache/dnf/copr:copr.fedorainfracloud.org:ceph:python3-asyncssh-5dfbfa110d1e0dad/packages/python3-asyncssh-2.7.0-1.el8.noarch.rpm: Header V3 RSA/SHA1 Signature, key ID 45bf1189: NOKEY 2021-09-02T07:26:37.256 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-7401.g9366f375.el8.noarch 4/4 2021-09-02T07:26:37.256 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-7401.g9366f375.el8.noarch 1/4 2021-09-02T07:26:37.257 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 2/4 2021-09-02T07:26:37.257 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/4 2021-09-02T07:26:37.529 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-asyncssh-2.7.0-1.el8.noarch 4/4 2021-09-02T07:26:37.529 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:37.530 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2021-09-02T07:26:37.530 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-cephadm-2:17.0.0-7401.g9366f375.el8.noarch 2021-09-02T07:26:37.530 INFO:teuthology.orchestra.run.smithi175.stdout: python3-asyncssh-2.7.0-1.el8.noarch 2021-09-02T07:26:37.530 INFO:teuthology.orchestra.run.smithi175.stdout: python3-babel-2.5.1-5.el8.noarch 2021-09-02T07:26:37.531 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-09-02T07:26:37.531 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:37.531 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:26:37.596 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-fuse 2021-09-02T07:26:38.044 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:45 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:38.045 INFO:teuthology.orchestra.run.smithi197.stdout:Copr repo for python3-asyncssh owned by ceph 1.2 kB/s | 1.0 kB 00:00 2021-09-02T07:26:38.150 INFO:teuthology.orchestra.run.smithi197.stderr:Importing GPG key 0x45BF1189: 2021-09-02T07:26:38.151 INFO:teuthology.orchestra.run.smithi197.stderr: Userid : "ceph_python3-asyncssh (None) " 2021-09-02T07:26:38.151 INFO:teuthology.orchestra.run.smithi197.stderr: Fingerprint: FD53 FC11 9FA9 2C47 E35D 9C35 339B 8950 45BF 1189 2021-09-02T07:26:38.151 INFO:teuthology.orchestra.run.smithi197.stderr: From : https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/pubkey.gpg 2021-09-02T07:26:38.267 INFO:teuthology.orchestra.run.smithi197.stdout:Key imported successfully 2021-09-02T07:26:38.268 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction check 2021-09-02T07:26:38.277 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction check succeeded. 2021-09-02T07:26:38.278 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction test 2021-09-02T07:26:38.365 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction test succeeded. 2021-09-02T07:26:38.368 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction 2021-09-02T07:26:38.947 INFO:teuthology.orchestra.run.smithi197.stdout: Preparing : 1/1 2021-09-02T07:26:39.029 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:26:39.030 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:26:39.031 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2021-09-02T07:26:39.031 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:26:39.031 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2021-09-02T07:26:39.031 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-fuse x86_64 2:17.0.0-7401.g9366f375.el8 ceph 868 k 2021-09-02T07:26:39.032 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2021-09-02T07:26:39.032 INFO:teuthology.orchestra.run.smithi175.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-09-02T07:26:39.032 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:39.032 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2021-09-02T07:26:39.032 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:26:39.033 INFO:teuthology.orchestra.run.smithi175.stdout:Install 2 Packages 2021-09-02T07:26:39.033 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:39.033 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 950 k 2021-09-02T07:26:39.034 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 3.1 M 2021-09-02T07:26:39.034 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2021-09-02T07:26:39.064 INFO:teuthology.orchestra.run.smithi175.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.5 MB/s | 82 kB 00:00 2021-09-02T07:26:39.878 INFO:teuthology.orchestra.run.smithi175.stdout:(2/2): ceph-fuse-17.0.0-7401.g9366f375.el8.x86_ 1.0 MB/s | 868 kB 00:00 2021-09-02T07:26:39.880 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:26:39.880 INFO:teuthology.orchestra.run.smithi175.stdout:Total 1.1 MB/s | 950 kB 00:00 2021-09-02T07:26:39.881 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2021-09-02T07:26:39.903 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2021-09-02T07:26:39.904 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2021-09-02T07:26:39.942 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-asyncssh-2.7.0-1.el8.noarch 1/4 2021-09-02T07:26:39.982 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2021-09-02T07:26:39.985 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2021-09-02T07:26:40.138 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2021-09-02T07:26:40.301 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 2/4 2021-09-02T07:26:40.419 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-09-02T07:26:40.523 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 3/4 2021-09-02T07:26:40.644 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-fuse-2:17.0.0-7401.g9366f375.el8.x86_64 2/2 2021-09-02T07:26:40.724 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-7401.g9366f375.el8.noarch 4/4 2021-09-02T07:26:40.964 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-7401.g9366f375.el8.noarch 4/4 2021-09-02T07:26:40.965 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-7401.g9366f375.el8.noarch 1/4 2021-09-02T07:26:40.965 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 2/4 2021-09-02T07:26:40.966 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/4 2021-09-02T07:26:40.992 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-fuse-2:17.0.0-7401.g9366f375.el8.x86_64 2/2 2021-09-02T07:26:40.993 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-fuse-2:17.0.0-7401.g9366f375.el8.x86_64 1/2 2021-09-02T07:26:41.241 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : python3-asyncssh-2.7.0-1.el8.noarch 4/4 2021-09-02T07:26:41.241 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:41.241 INFO:teuthology.orchestra.run.smithi197.stdout:Installed: 2021-09-02T07:26:41.242 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-mgr-cephadm-2:17.0.0-7401.g9366f375.el8.noarch 2021-09-02T07:26:41.242 INFO:teuthology.orchestra.run.smithi197.stdout: python3-asyncssh-2.7.0-1.el8.noarch 2021-09-02T07:26:41.242 INFO:teuthology.orchestra.run.smithi197.stdout: python3-babel-2.5.1-5.el8.noarch 2021-09-02T07:26:41.242 INFO:teuthology.orchestra.run.smithi197.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-09-02T07:26:41.243 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:41.243 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:26:41.271 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-09-02T07:26:41.271 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:41.271 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2021-09-02T07:26:41.272 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-fuse-2:17.0.0-7401.g9366f375.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-09-02T07:26:41.272 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:41.272 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:26:41.315 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install ceph-fuse 2021-09-02T07:26:41.513 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install librados-devel 2021-09-02T07:26:41.759 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:01:48 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:41.967 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:48 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:42.716 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:26:43.045 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:26:43.045 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:26:43.045 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repo Size 2021-09-02T07:26:43.046 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:26:43.046 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2021-09-02T07:26:43.046 INFO:teuthology.orchestra.run.smithi175.stdout: librados-devel x86_64 2:17.0.0-7401.g9366f375.el8 ceph 125 k 2021-09-02T07:26:43.046 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:43.047 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2021-09-02T07:26:43.047 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:26:43.047 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2021-09-02T07:26:43.047 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:43.047 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 125 k 2021-09-02T07:26:43.048 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 432 k 2021-09-02T07:26:43.048 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2021-09-02T07:26:43.049 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:26:43.049 INFO:teuthology.orchestra.run.smithi197.stdout: Package Arch Version Repository Size 2021-09-02T07:26:43.049 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:26:43.050 INFO:teuthology.orchestra.run.smithi197.stdout:Installing: 2021-09-02T07:26:43.050 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-fuse x86_64 2:17.0.0-7401.g9366f375.el8 ceph 868 k 2021-09-02T07:26:43.050 INFO:teuthology.orchestra.run.smithi197.stdout:Installing dependencies: 2021-09-02T07:26:43.050 INFO:teuthology.orchestra.run.smithi197.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-09-02T07:26:43.051 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:43.051 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction Summary 2021-09-02T07:26:43.051 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:26:43.051 INFO:teuthology.orchestra.run.smithi197.stdout:Install 2 Packages 2021-09-02T07:26:43.052 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:43.052 INFO:teuthology.orchestra.run.smithi197.stdout:Total download size: 950 k 2021-09-02T07:26:43.052 INFO:teuthology.orchestra.run.smithi197.stdout:Installed size: 3.1 M 2021-09-02T07:26:43.052 INFO:teuthology.orchestra.run.smithi197.stdout:Downloading Packages: 2021-09-02T07:26:43.052 INFO:teuthology.orchestra.run.smithi197.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.1 MB/s | 82 kB 00:00 2021-09-02T07:26:43.053 INFO:teuthology.orchestra.run.smithi197.stdout:(2/2): ceph-fuse-17.0.0-7401.g9366f375.el8.x86_ 4.0 MB/s | 868 kB 00:00 2021-09-02T07:26:43.053 INFO:teuthology.orchestra.run.smithi197.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:26:43.053 INFO:teuthology.orchestra.run.smithi197.stdout:Total 4.3 MB/s | 950 kB 00:00 2021-09-02T07:26:43.053 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction check 2021-09-02T07:26:43.053 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction check succeeded. 2021-09-02T07:26:43.054 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction test 2021-09-02T07:26:43.054 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction test succeeded. 2021-09-02T07:26:43.054 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction 2021-09-02T07:26:43.090 INFO:teuthology.orchestra.run.smithi175.stdout:librados-devel-17.0.0-7401.g9366f375.el8.x86_64 820 kB/s | 125 kB 00:00 2021-09-02T07:26:43.091 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:26:43.092 INFO:teuthology.orchestra.run.smithi175.stdout:Total 813 kB/s | 125 kB 00:00 2021-09-02T07:26:43.092 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2021-09-02T07:26:43.098 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2021-09-02T07:26:43.098 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2021-09-02T07:26:43.148 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2021-09-02T07:26:43.151 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2021-09-02T07:26:43.210 INFO:teuthology.orchestra.run.smithi197.stdout: Preparing : 1/1 2021-09-02T07:26:43.281 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2021-09-02T07:26:43.459 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librados-devel-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:26:43.497 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-09-02T07:26:43.709 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librados-devel-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:26:43.732 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : ceph-fuse-2:17.0.0-7401.g9366f375.el8.x86_64 2/2 2021-09-02T07:26:44.003 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados-devel-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:26:44.004 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:44.004 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2021-09-02T07:26:44.004 INFO:teuthology.orchestra.run.smithi175.stdout: librados-devel-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:26:44.004 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:44.004 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:26:44.073 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: ceph-fuse-2:17.0.0-7401.g9366f375.el8.x86_64 2/2 2021-09-02T07:26:44.073 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : ceph-fuse-2:17.0.0-7401.g9366f375.el8.x86_64 1/2 2021-09-02T07:26:44.182 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install libcephfs2 2021-09-02T07:26:44.357 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-09-02T07:26:44.357 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:44.358 INFO:teuthology.orchestra.run.smithi197.stdout:Installed: 2021-09-02T07:26:44.358 INFO:teuthology.orchestra.run.smithi197.stdout: ceph-fuse-2:17.0.0-7401.g9366f375.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-09-02T07:26:44.358 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:44.358 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:26:44.568 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install librados-devel 2021-09-02T07:26:44.633 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:51 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:45.026 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:01:52 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:45.327 INFO:teuthology.orchestra.run.smithi175.stdout:Package libcephfs2-2:17.0.0-7401.g9366f375.el8.x86_64 is already installed. 2021-09-02T07:26:45.579 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:26:45.580 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2021-09-02T07:26:45.580 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:26:45.630 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install libcephfs-devel 2021-09-02T07:26:45.968 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:26:45.969 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:26:45.969 INFO:teuthology.orchestra.run.smithi197.stdout: Package Arch Version Repo Size 2021-09-02T07:26:45.969 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:26:45.970 INFO:teuthology.orchestra.run.smithi197.stdout:Installing: 2021-09-02T07:26:45.970 INFO:teuthology.orchestra.run.smithi197.stdout: librados-devel x86_64 2:17.0.0-7401.g9366f375.el8 ceph 125 k 2021-09-02T07:26:45.970 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:45.970 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction Summary 2021-09-02T07:26:45.971 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:26:45.971 INFO:teuthology.orchestra.run.smithi197.stdout:Install 1 Package 2021-09-02T07:26:45.971 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:45.971 INFO:teuthology.orchestra.run.smithi197.stdout:Total download size: 125 k 2021-09-02T07:26:45.971 INFO:teuthology.orchestra.run.smithi197.stdout:Installed size: 432 k 2021-09-02T07:26:45.972 INFO:teuthology.orchestra.run.smithi197.stdout:Downloading Packages: 2021-09-02T07:26:46.078 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:53 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:46.122 INFO:teuthology.orchestra.run.smithi197.stdout:librados-devel-17.0.0-7401.g9366f375.el8.x86_64 816 kB/s | 125 kB 00:00 2021-09-02T07:26:46.123 INFO:teuthology.orchestra.run.smithi197.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:26:46.124 INFO:teuthology.orchestra.run.smithi197.stdout:Total 809 kB/s | 125 kB 00:00 2021-09-02T07:26:46.124 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction check 2021-09-02T07:26:46.129 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction check succeeded. 2021-09-02T07:26:46.130 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction test 2021-09-02T07:26:46.181 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction test succeeded. 2021-09-02T07:26:46.184 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction 2021-09-02T07:26:46.297 INFO:teuthology.orchestra.run.smithi197.stdout: Preparing : 1/1 2021-09-02T07:26:46.481 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : librados-devel-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:26:46.722 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: librados-devel-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:26:47.010 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : librados-devel-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:26:47.011 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:47.011 INFO:teuthology.orchestra.run.smithi197.stdout:Installed: 2021-09-02T07:26:47.011 INFO:teuthology.orchestra.run.smithi197.stdout: librados-devel-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:26:47.012 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:47.013 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:26:47.067 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:26:47.068 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:26:47.069 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repo Size 2021-09-02T07:26:47.069 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:26:47.069 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2021-09-02T07:26:47.069 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs-devel x86_64 2:17.0.0-7401.g9366f375.el8 ceph 24 k 2021-09-02T07:26:47.070 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:47.070 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2021-09-02T07:26:47.070 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:26:47.070 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2021-09-02T07:26:47.071 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:47.071 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 24 k 2021-09-02T07:26:47.071 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 102 k 2021-09-02T07:26:47.071 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2021-09-02T07:26:47.181 INFO:teuthology.orchestra.run.smithi175.stdout:libcephfs-devel-17.0.0-7401.g9366f375.el8.x86_6 212 kB/s | 24 kB 00:00 2021-09-02T07:26:47.182 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:26:47.182 INFO:teuthology.orchestra.run.smithi175.stdout:Total 210 kB/s | 24 kB 00:00 2021-09-02T07:26:47.183 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2021-09-02T07:26:47.185 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install libcephfs2 2021-09-02T07:26:47.186 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2021-09-02T07:26:47.187 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2021-09-02T07:26:47.187 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2021-09-02T07:26:47.190 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2021-09-02T07:26:47.240 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2021-09-02T07:26:47.409 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephfs-devel-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:26:47.630 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:26:47.638 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:01:54 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:48.371 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephfs-devel-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:26:48.371 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:48.372 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2021-09-02T07:26:48.372 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs-devel-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:26:48.372 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:48.372 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:26:48.373 INFO:teuthology.orchestra.run.smithi197.stdout:Package libcephfs2-2:17.0.0-7401.g9366f375.el8.x86_64 is already installed. 2021-09-02T07:26:48.439 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install librados2 2021-09-02T07:26:48.625 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:26:48.626 INFO:teuthology.orchestra.run.smithi197.stdout:Nothing to do. 2021-09-02T07:26:48.626 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:26:48.689 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install libcephfs-devel 2021-09-02T07:26:48.895 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:55 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:49.139 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:01:56 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:49.618 INFO:teuthology.orchestra.run.smithi175.stdout:Package librados2-2:17.0.0-7401.g9366f375.el8.x86_64 is already installed. 2021-09-02T07:26:49.867 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:26:49.868 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2021-09-02T07:26:49.868 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:26:49.919 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install librbd1 2021-09-02T07:26:50.109 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:26:50.110 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:26:50.110 INFO:teuthology.orchestra.run.smithi197.stdout: Package Arch Version Repo Size 2021-09-02T07:26:50.110 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:26:50.111 INFO:teuthology.orchestra.run.smithi197.stdout:Installing: 2021-09-02T07:26:50.111 INFO:teuthology.orchestra.run.smithi197.stdout: libcephfs-devel x86_64 2:17.0.0-7401.g9366f375.el8 ceph 24 k 2021-09-02T07:26:50.111 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:50.111 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction Summary 2021-09-02T07:26:50.112 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:26:50.112 INFO:teuthology.orchestra.run.smithi197.stdout:Install 1 Package 2021-09-02T07:26:50.112 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:50.112 INFO:teuthology.orchestra.run.smithi197.stdout:Total download size: 24 k 2021-09-02T07:26:50.113 INFO:teuthology.orchestra.run.smithi197.stdout:Installed size: 102 k 2021-09-02T07:26:50.113 INFO:teuthology.orchestra.run.smithi197.stdout:Downloading Packages: 2021-09-02T07:26:50.221 INFO:teuthology.orchestra.run.smithi197.stdout:libcephfs-devel-17.0.0-7401.g9366f375.el8.x86_6 217 kB/s | 24 kB 00:00 2021-09-02T07:26:50.222 INFO:teuthology.orchestra.run.smithi197.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:26:50.222 INFO:teuthology.orchestra.run.smithi197.stdout:Total 214 kB/s | 24 kB 00:00 2021-09-02T07:26:50.222 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction check 2021-09-02T07:26:50.225 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction check succeeded. 2021-09-02T07:26:50.225 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction test 2021-09-02T07:26:50.227 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction test succeeded. 2021-09-02T07:26:50.230 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction 2021-09-02T07:26:50.305 INFO:teuthology.orchestra.run.smithi197.stdout: Preparing : 1/1 2021-09-02T07:26:50.365 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:57 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:50.472 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : libcephfs-devel-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:26:50.676 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:26:50.931 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : libcephfs-devel-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:26:50.931 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:50.931 INFO:teuthology.orchestra.run.smithi197.stdout:Installed: 2021-09-02T07:26:50.932 INFO:teuthology.orchestra.run.smithi197.stdout: libcephfs-devel-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:26:50.932 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:26:50.932 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:26:51.003 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install librados2 2021-09-02T07:26:51.093 INFO:teuthology.orchestra.run.smithi175.stdout:Package librbd1-2:17.0.0-7401.g9366f375.el8.x86_64 is already installed. 2021-09-02T07:26:51.341 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:26:51.342 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2021-09-02T07:26:51.342 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:26:51.397 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install python3-rados 2021-09-02T07:26:51.450 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:01:58 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:51.842 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:58 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:52.146 INFO:teuthology.orchestra.run.smithi197.stdout:Package librados2-2:17.0.0-7401.g9366f375.el8.x86_64 is already installed. 2021-09-02T07:26:52.395 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:26:52.396 INFO:teuthology.orchestra.run.smithi197.stdout:Nothing to do. 2021-09-02T07:26:52.396 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:26:52.459 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install librbd1 2021-09-02T07:26:52.551 INFO:teuthology.orchestra.run.smithi175.stdout:Package python3-rados-2:17.0.0-7401.g9366f375.el8.x86_64 is already installed. 2021-09-02T07:26:52.806 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:26:52.807 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2021-09-02T07:26:52.807 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:26:52.863 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install python3-rgw 2021-09-02T07:26:52.906 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:01:59 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:53.311 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:02:00 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:53.596 INFO:teuthology.orchestra.run.smithi197.stdout:Package librbd1-2:17.0.0-7401.g9366f375.el8.x86_64 is already installed. 2021-09-02T07:26:53.853 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:26:53.854 INFO:teuthology.orchestra.run.smithi197.stdout:Nothing to do. 2021-09-02T07:26:53.854 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:26:53.912 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install python3-rados 2021-09-02T07:26:54.360 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:02:01 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:54.439 INFO:teuthology.orchestra.run.smithi175.stdout:Package python3-rgw-2:17.0.0-7401.g9366f375.el8.x86_64 is already installed. 2021-09-02T07:26:54.688 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:26:54.689 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2021-09-02T07:26:54.689 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:26:55.044 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install python3-cephfs 2021-09-02T07:26:55.497 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:02:02 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:55.497 INFO:teuthology.orchestra.run.smithi197.stdout:Package python3-rados-2:17.0.0-7401.g9366f375.el8.x86_64 is already installed. 2021-09-02T07:26:55.745 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:26:55.746 INFO:teuthology.orchestra.run.smithi197.stdout:Nothing to do. 2021-09-02T07:26:55.746 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:26:55.800 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install python3-rgw 2021-09-02T07:26:56.208 INFO:teuthology.orchestra.run.smithi175.stdout:Package python3-cephfs-2:17.0.0-7401.g9366f375.el8.x86_64 is already installed. 2021-09-02T07:26:56.247 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:02:03 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:56.455 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:26:56.456 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2021-09-02T07:26:56.456 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:26:56.502 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install python3-rbd 2021-09-02T07:26:56.950 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:02:03 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:56.962 INFO:teuthology.orchestra.run.smithi197.stdout:Package python3-rgw-2:17.0.0-7401.g9366f375.el8.x86_64 is already installed. 2021-09-02T07:26:57.215 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:26:57.217 INFO:teuthology.orchestra.run.smithi197.stdout:Nothing to do. 2021-09-02T07:26:57.217 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:26:57.286 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install python3-cephfs 2021-09-02T07:26:57.683 INFO:teuthology.orchestra.run.smithi175.stdout:Package python3-rbd-2:17.0.0-7401.g9366f375.el8.x86_64 is already installed. 2021-09-02T07:26:57.731 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:02:04 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:57.932 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:26:57.932 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2021-09-02T07:26:57.933 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:26:57.987 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install rbd-fuse 2021-09-02T07:26:58.429 INFO:teuthology.orchestra.run.smithi197.stdout:Package python3-cephfs-2:17.0.0-7401.g9366f375.el8.x86_64 is already installed. 2021-09-02T07:26:58.440 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:02:05 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:58.676 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:26:58.677 INFO:teuthology.orchestra.run.smithi197.stdout:Nothing to do. 2021-09-02T07:26:58.677 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:26:58.744 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install python3-rbd 2021-09-02T07:26:59.194 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:02:06 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:26:59.397 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:26:59.398 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:26:59.399 INFO:teuthology.orchestra.run.smithi175.stdout: Package Architecture Version Repository Size 2021-09-02T07:26:59.399 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:26:59.400 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2021-09-02T07:26:59.400 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-fuse x86_64 2:17.0.0-7401.g9366f375.el8 ceph 94 k 2021-09-02T07:26:59.400 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:59.401 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2021-09-02T07:26:59.401 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:26:59.402 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2021-09-02T07:26:59.402 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:26:59.403 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 94 k 2021-09-02T07:26:59.403 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 281 k 2021-09-02T07:26:59.404 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2021-09-02T07:26:59.577 INFO:teuthology.orchestra.run.smithi175.stdout:rbd-fuse-17.0.0-7401.g9366f375.el8.x86_64.rpm 527 kB/s | 94 kB 00:00 2021-09-02T07:26:59.578 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:26:59.578 INFO:teuthology.orchestra.run.smithi175.stdout:Total 523 kB/s | 94 kB 00:00 2021-09-02T07:26:59.578 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2021-09-02T07:26:59.588 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2021-09-02T07:26:59.589 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2021-09-02T07:26:59.638 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2021-09-02T07:26:59.641 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2021-09-02T07:26:59.764 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2021-09-02T07:26:59.895 INFO:teuthology.orchestra.run.smithi197.stdout:Package python3-rbd-2:17.0.0-7401.g9366f375.el8.x86_64 is already installed. 2021-09-02T07:26:59.956 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-fuse-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:27:00.142 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:27:00.143 INFO:teuthology.orchestra.run.smithi197.stdout:Nothing to do. 2021-09-02T07:27:00.143 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:27:00.188 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: rbd-fuse-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:27:00.211 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install rbd-fuse 2021-09-02T07:27:00.469 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-fuse-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:27:00.469 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:00.470 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2021-09-02T07:27:00.470 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-fuse-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:27:00.471 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:00.471 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:27:00.662 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:02:07 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:27:01.096 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install rbd-mirror 2021-09-02T07:27:01.550 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:02:08 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:27:01.621 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:27:01.623 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:27:01.623 INFO:teuthology.orchestra.run.smithi197.stdout: Package Architecture Version Repository Size 2021-09-02T07:27:01.623 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:27:01.624 INFO:teuthology.orchestra.run.smithi197.stdout:Installing: 2021-09-02T07:27:01.624 INFO:teuthology.orchestra.run.smithi197.stdout: rbd-fuse x86_64 2:17.0.0-7401.g9366f375.el8 ceph 94 k 2021-09-02T07:27:01.624 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:01.624 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction Summary 2021-09-02T07:27:01.625 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:27:01.625 INFO:teuthology.orchestra.run.smithi197.stdout:Install 1 Package 2021-09-02T07:27:01.625 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:01.625 INFO:teuthology.orchestra.run.smithi197.stdout:Total download size: 94 k 2021-09-02T07:27:01.625 INFO:teuthology.orchestra.run.smithi197.stdout:Installed size: 281 k 2021-09-02T07:27:01.626 INFO:teuthology.orchestra.run.smithi197.stdout:Downloading Packages: 2021-09-02T07:27:01.755 INFO:teuthology.orchestra.run.smithi197.stdout:rbd-fuse-17.0.0-7401.g9366f375.el8.x86_64.rpm 712 kB/s | 94 kB 00:00 2021-09-02T07:27:01.756 INFO:teuthology.orchestra.run.smithi197.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:27:01.756 INFO:teuthology.orchestra.run.smithi197.stdout:Total 705 kB/s | 94 kB 00:00 2021-09-02T07:27:01.756 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction check 2021-09-02T07:27:01.767 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction check succeeded. 2021-09-02T07:27:01.767 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction test 2021-09-02T07:27:01.816 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction test succeeded. 2021-09-02T07:27:01.819 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction 2021-09-02T07:27:01.935 INFO:teuthology.orchestra.run.smithi197.stdout: Preparing : 1/1 2021-09-02T07:27:02.112 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : rbd-fuse-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:27:02.341 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: rbd-fuse-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:27:02.506 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:27:02.507 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:27:02.507 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repo Size 2021-09-02T07:27:02.508 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:27:02.508 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2021-09-02T07:27:02.508 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-mirror x86_64 2:17.0.0-7401.g9366f375.el8 ceph 3.8 M 2021-09-02T07:27:02.508 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:02.509 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2021-09-02T07:27:02.509 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:27:02.509 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2021-09-02T07:27:02.509 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:02.510 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 3.8 M 2021-09-02T07:27:02.510 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 19 M 2021-09-02T07:27:02.510 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2021-09-02T07:27:02.646 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : rbd-fuse-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:27:02.646 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:02.647 INFO:teuthology.orchestra.run.smithi197.stdout:Installed: 2021-09-02T07:27:02.647 INFO:teuthology.orchestra.run.smithi197.stdout: rbd-fuse-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:27:02.647 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:02.647 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:27:02.833 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install rbd-mirror 2021-09-02T07:27:03.038 INFO:teuthology.orchestra.run.smithi175.stdout:rbd-mirror-17.0.0-7401.g9366f375.el8.x86_64.rpm 7.1 MB/s | 3.8 MB 00:00 2021-09-02T07:27:03.039 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:27:03.039 INFO:teuthology.orchestra.run.smithi175.stdout:Total 7.1 MB/s | 3.8 MB 00:00 2021-09-02T07:27:03.040 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2021-09-02T07:27:03.052 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2021-09-02T07:27:03.052 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2021-09-02T07:27:03.115 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2021-09-02T07:27:03.118 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2021-09-02T07:27:03.283 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:02:10 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:27:03.592 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2021-09-02T07:27:03.782 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-mirror-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:27:04.181 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: rbd-mirror-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:27:04.242 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:27:04.243 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:27:04.243 INFO:teuthology.orchestra.run.smithi197.stdout: Package Arch Version Repo Size 2021-09-02T07:27:04.243 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:27:04.243 INFO:teuthology.orchestra.run.smithi197.stdout:Installing: 2021-09-02T07:27:04.244 INFO:teuthology.orchestra.run.smithi197.stdout: rbd-mirror x86_64 2:17.0.0-7401.g9366f375.el8 ceph 3.8 M 2021-09-02T07:27:04.244 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:04.244 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction Summary 2021-09-02T07:27:04.244 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:27:04.245 INFO:teuthology.orchestra.run.smithi197.stdout:Install 1 Package 2021-09-02T07:27:04.245 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:04.245 INFO:teuthology.orchestra.run.smithi197.stdout:Total download size: 3.8 M 2021-09-02T07:27:04.245 INFO:teuthology.orchestra.run.smithi197.stdout:Installed size: 19 M 2021-09-02T07:27:04.246 INFO:teuthology.orchestra.run.smithi197.stdout:Downloading Packages: 2021-09-02T07:27:04.433 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-mirror-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:27:04.434 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:04.434 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2021-09-02T07:27:04.434 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-mirror-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:27:04.434 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:04.435 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:27:04.561 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install rbd-nbd 2021-09-02T07:27:04.731 INFO:teuthology.orchestra.run.smithi197.stdout:rbd-mirror-17.0.0-7401.g9366f375.el8.x86_64.rpm 7.7 MB/s | 3.8 MB 00:00 2021-09-02T07:27:04.731 INFO:teuthology.orchestra.run.smithi197.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:27:04.732 INFO:teuthology.orchestra.run.smithi197.stdout:Total 7.7 MB/s | 3.8 MB 00:00 2021-09-02T07:27:04.733 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction check 2021-09-02T07:27:04.745 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction check succeeded. 2021-09-02T07:27:04.746 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction test 2021-09-02T07:27:04.809 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction test succeeded. 2021-09-02T07:27:04.812 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction 2021-09-02T07:27:05.013 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:02:12 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:27:05.282 INFO:teuthology.orchestra.run.smithi197.stdout: Preparing : 1/1 2021-09-02T07:27:05.454 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : rbd-mirror-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:27:05.806 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: rbd-mirror-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:27:05.979 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2021-09-02T07:27:05.980 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:27:05.980 INFO:teuthology.orchestra.run.smithi175.stdout: Package Architecture Version Repository Size 2021-09-02T07:27:05.980 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:27:05.981 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2021-09-02T07:27:05.981 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-nbd x86_64 2:17.0.0-7401.g9366f375.el8 ceph 173 k 2021-09-02T07:27:05.981 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:05.981 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2021-09-02T07:27:05.982 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2021-09-02T07:27:05.982 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2021-09-02T07:27:05.982 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:05.983 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 173 k 2021-09-02T07:27:05.983 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 563 k 2021-09-02T07:27:05.983 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2021-09-02T07:27:06.153 INFO:teuthology.orchestra.run.smithi175.stdout:rbd-nbd-17.0.0-7401.g9366f375.el8.x86_64.rpm 1.0 MB/s | 173 kB 00:00 2021-09-02T07:27:06.153 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:27:06.154 INFO:teuthology.orchestra.run.smithi175.stdout:Total 998 kB/s | 173 kB 00:00 2021-09-02T07:27:06.154 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2021-09-02T07:27:06.167 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2021-09-02T07:27:06.167 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2021-09-02T07:27:06.216 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2021-09-02T07:27:06.219 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2021-09-02T07:27:06.264 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : rbd-mirror-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:27:06.265 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:06.265 INFO:teuthology.orchestra.run.smithi197.stdout:Installed: 2021-09-02T07:27:06.266 INFO:teuthology.orchestra.run.smithi197.stdout: rbd-mirror-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:27:06.266 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:06.266 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:27:06.433 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2021-09-02T07:27:06.656 DEBUG:teuthology.orchestra.run.smithi197:> sudo yum -y install rbd-nbd 2021-09-02T07:27:06.671 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-nbd-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:27:06.909 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: rbd-nbd-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:27:07.114 INFO:teuthology.orchestra.run.smithi197.stdout:Last metadata expiration check: 0:02:14 ago on Thu 02 Sep 2021 07:24:53 AM UTC. 2021-09-02T07:27:07.190 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-nbd-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:27:07.191 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:07.191 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2021-09-02T07:27:07.191 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-nbd-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:27:07.191 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:07.191 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2021-09-02T07:27:07.311 DEBUG:teuthology.parallel:result is None 2021-09-02T07:27:08.057 INFO:teuthology.orchestra.run.smithi197.stdout:Dependencies resolved. 2021-09-02T07:27:08.058 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:27:08.058 INFO:teuthology.orchestra.run.smithi197.stdout: Package Architecture Version Repository Size 2021-09-02T07:27:08.058 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:27:08.059 INFO:teuthology.orchestra.run.smithi197.stdout:Installing: 2021-09-02T07:27:08.059 INFO:teuthology.orchestra.run.smithi197.stdout: rbd-nbd x86_64 2:17.0.0-7401.g9366f375.el8 ceph 173 k 2021-09-02T07:27:08.059 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:08.059 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction Summary 2021-09-02T07:27:08.059 INFO:teuthology.orchestra.run.smithi197.stdout:================================================================================ 2021-09-02T07:27:08.060 INFO:teuthology.orchestra.run.smithi197.stdout:Install 1 Package 2021-09-02T07:27:08.060 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:08.060 INFO:teuthology.orchestra.run.smithi197.stdout:Total download size: 173 k 2021-09-02T07:27:08.060 INFO:teuthology.orchestra.run.smithi197.stdout:Installed size: 563 k 2021-09-02T07:27:08.061 INFO:teuthology.orchestra.run.smithi197.stdout:Downloading Packages: 2021-09-02T07:27:08.211 INFO:teuthology.orchestra.run.smithi197.stdout:rbd-nbd-17.0.0-7401.g9366f375.el8.x86_64.rpm 1.1 MB/s | 173 kB 00:00 2021-09-02T07:27:08.212 INFO:teuthology.orchestra.run.smithi197.stdout:-------------------------------------------------------------------------------- 2021-09-02T07:27:08.212 INFO:teuthology.orchestra.run.smithi197.stdout:Total 1.1 MB/s | 173 kB 00:00 2021-09-02T07:27:08.212 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction check 2021-09-02T07:27:08.224 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction check succeeded. 2021-09-02T07:27:08.224 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction test 2021-09-02T07:27:08.273 INFO:teuthology.orchestra.run.smithi197.stdout:Transaction test succeeded. 2021-09-02T07:27:08.276 INFO:teuthology.orchestra.run.smithi197.stdout:Running transaction 2021-09-02T07:27:08.403 INFO:teuthology.orchestra.run.smithi197.stdout: Preparing : 1/1 2021-09-02T07:27:08.598 INFO:teuthology.orchestra.run.smithi197.stdout: Installing : rbd-nbd-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:27:08.841 INFO:teuthology.orchestra.run.smithi197.stdout: Running scriptlet: rbd-nbd-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:27:09.140 INFO:teuthology.orchestra.run.smithi197.stdout: Verifying : rbd-nbd-2:17.0.0-7401.g9366f375.el8.x86_64 1/1 2021-09-02T07:27:09.141 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:09.142 INFO:teuthology.orchestra.run.smithi197.stdout:Installed: 2021-09-02T07:27:09.142 INFO:teuthology.orchestra.run.smithi197.stdout: rbd-nbd-2:17.0.0-7401.g9366f375.el8.x86_64 2021-09-02T07:27:09.142 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:09.143 INFO:teuthology.orchestra.run.smithi197.stdout:Complete! 2021-09-02T07:27:09.301 DEBUG:teuthology.parallel:result is None 2021-09-02T07:27:09.302 DEBUG:teuthology.orchestra.run.smithi175:> sudo lsb_release -is 2021-09-02T07:27:09.343 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS 2021-09-02T07:27:09.345 DEBUG:teuthology.misc:System to be installed: CentOS 2021-09-02T07:27:09.346 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9366f375ac31713c35e559edbd160565b784db1d 2021-09-02T07:27:09.599 DEBUG:teuthology.orchestra.run.smithi175:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-09-02T07:27:09.622 INFO:teuthology.orchestra.run.smithi175.stdout:17.0.0-7401.g9366f375.el8 2021-09-02T07:27:09.623 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-7401.g9366f375.el8 2021-09-02T07:27:09.624 INFO:teuthology.task.install:The correct ceph version 17.0.0-7401.g9366f375 is installed. 2021-09-02T07:27:09.624 DEBUG:teuthology.orchestra.run.smithi197:> sudo lsb_release -is 2021-09-02T07:27:09.658 INFO:teuthology.orchestra.run.smithi197.stdout:CentOS 2021-09-02T07:27:09.659 DEBUG:teuthology.misc:System to be installed: CentOS 2021-09-02T07:27:09.660 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=9366f375ac31713c35e559edbd160565b784db1d 2021-09-02T07:27:09.808 DEBUG:teuthology.orchestra.run.smithi197:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-09-02T07:27:09.826 INFO:teuthology.orchestra.run.smithi197.stdout:17.0.0-7401.g9366f375.el8 2021-09-02T07:27:09.827 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-7401.g9366f375.el8 2021-09-02T07:27:09.827 INFO:teuthology.task.install:The correct ceph version 17.0.0-7401.g9366f375 is installed. 2021-09-02T07:27:09.828 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-09-02T07:27:09.828 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:27:09.829 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-09-02T07:27:09.857 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:27:09.857 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-09-02T07:27:09.890 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-09-02T07:27:09.891 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:27:09.891 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/usr/bin/daemon-helper 2021-09-02T07:27:09.924 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-09-02T07:27:09.991 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:27:09.992 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd of=/usr/bin/daemon-helper 2021-09-02T07:27:10.015 DEBUG:teuthology.orchestra.run.smithi197:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-09-02T07:27:10.078 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-09-02T07:27:10.079 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:27:10.079 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/usr/bin/adjust-ulimits 2021-09-02T07:27:10.105 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-09-02T07:27:10.172 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:27:10.173 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd of=/usr/bin/adjust-ulimits 2021-09-02T07:27:10.196 DEBUG:teuthology.orchestra.run.smithi197:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-09-02T07:27:10.258 INFO:teuthology.run_tasks:Running task ceph... 2021-09-02T07:27:10.328 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-09-02T07:27:10.329 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 777 /var/log/ceph 2021-09-02T07:27:10.331 DEBUG:teuthology.orchestra.run.smithi197:> sudo chmod 777 /var/log/ceph 2021-09-02T07:27:10.355 INFO:tasks.ceph:Disabling ceph logrotate... 2021-09-02T07:27:10.355 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f -- /etc/logrotate.d/ceph 2021-09-02T07:27:10.397 DEBUG:teuthology.orchestra.run.smithi197:> sudo rm -f -- /etc/logrotate.d/ceph 2021-09-02T07:27:10.426 INFO:tasks.ceph:Creating extra log directories... 2021-09-02T07:27:10.426 DEBUG:teuthology.orchestra.run.smithi175:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-09-02T07:27:10.468 DEBUG:teuthology.orchestra.run.smithi197:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-09-02T07:27:10.534 INFO:tasks.ceph:Creating ceph cluster c1... 2021-09-02T07:27:10.534 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-09-02T07:27:10.535 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/soumyakoduri-2021-09-02_01:20:40-rgw-wip-skoduri-dbstore-object-distro-basic-smithi/6371861', 'branch': 'wip-skoduri-dbstore-object', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '6371861', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'soumyakoduri-2021-09-02_01:20:40-rgw-wip-skoduri-dbstore-object-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.3', 'overrides': {'admin_socket': {'branch': 'wip-skoduri-dbstore-object'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9366f375ac31713c35e559edbd160565b784db1d', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '9366f375ac31713c35e559edbd160565b784db1d'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-skoduri-dbstore-object', 'sha1': '9366f375ac31713c35e559edbd160565b784db1d'}}, 'owner': 'scheduled_soumyakoduri@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '9366f375ac31713c35e559edbd160565b784db1d', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-skoduri-dbstore-object', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_9366f375ac31713c35e559edbd160565b784db1d/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '9366f375ac31713c35e559edbd160565b784db1d', 'targets': {'smithi175.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtuWE9inORipzeE+c+qLMP7PU0ecZk/wcl3RE/RP6rStzOx64hb5BodiNDj49HIXSUwRgPDulx5ybgPNn7GbpZtR7Cf9V5PRoeIXwuW8aP6+Zl7H4zba6Xqh7fQ0w3f3yM45Mx9rPUFZtxNL/PS91bh5HU6ZAv1AUKBl3tQePOsxOBg2LJqFLov7K6bG71BDb6shOEaq6f+ZUaJE7+4HX2msifQnRmuq0OXWrpxAbTmqdAbAS9CP3Aof9f1w+6g12XTkTrB2UhRBiDLgjLj6osOXyXWetUMJ9WHCyfDOhtbavgpkjQWwDH5PWwbGAhHMmlXFD/siT2vevCswIWGaWdKimy2A1o/cMcS9v2slsiPeoiQWBbBkvO3rouamC3hUEwCZYbNduTOdnngQLfzxxJ2fiBFyXuCouLGL3eP06Bei1opwh9GIjq3iSMCJVA4OJeNw2jp0Zc22mUq3LO/dgsYFygus1flReNTmTlq4s0mAKoHYFPN3NkMLktBo62dW8=', 'smithi197.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5bB3TaCFQfuTSeKh7NnjyOp1hVDtEXsXv52fZPgcyTE20csjTrt9TJJJfjAYPwU4o3iukgEiMnem+NMjtU+9SJZ2PPGOY0MwzazJCuotZ54nUPf9OaJI8lu4NVq26s7PkEWXa8+pwCmZ9hMBt7Ww216VdaAUgectXUk93NVjZAC0Nd387dL1+iT3rwaRjelbeKwkHCxyOIs+k7J3oPkZ78HKR1IlSUwHok1tdNyOWa0L7cqwoGrDslBdZ3+QYXIyMZtX8LWlRimaAjyYc53aOBn81EecHmVoSG6CxWiA94ZW+x+2/ftcObopTJWSxNP1ywgY+JtDGLjxlLKJkKNtISZ8Zg4NZEm31xV3vmXeqihw+AdX+pmHKwZeC31bmkcN/2c0bVcScBrno8DsBL3C2CgC1d3xtvaw9VcUp85m/Q8m8h2NNxGVDuKcm/BtPIKqbl+ItW7629pp0YcgNka+VKwn17KWN0fFTl+ghKKXc9BE1ynWEE4VwL6xJjeU8kds='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9366f375ac31713c35e559edbd160565b784db1d', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2021-09-02_01:20:40', 'tube': 'smithi', 'user': 'soumyakoduri', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.30581'} 2021-09-02T07:27:10.536 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-09-02T07:27:10.538 DEBUG:teuthology.orchestra.run.smithi197:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-09-02T07:27:10.552 DEBUG:teuthology.orchestra.run.smithi175:> sudo install -d -m0777 -- /var/run/ceph 2021-09-02T07:27:10.593 DEBUG:teuthology.orchestra.run.smithi197:> sudo install -d -m0777 -- /var/run/ceph 2021-09-02T07:27:10.618 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:27:10.619 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/scratch_devs of=/dev/stdout 2021-09-02T07:27:10.672 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-09-02T07:27:10.673 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-09-02T07:27:10.726 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-09-02T07:27:10.727 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-02T07:27:10.727 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 6h/6d Inode: 110371 Links: 1 2021-09-02T07:27:10.727 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-02T07:27:10.727 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2021-09-02T07:27:10.728 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2021-09-02 07:27:03.979674341 +0000 2021-09-02T07:27:10.728 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2021-09-02 07:24:06.740523416 +0000 2021-09-02T07:27:10.728 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2021-09-02 07:24:06.740523416 +0000 2021-09-02T07:27:10.728 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: - 2021-09-02T07:27:10.737 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2021-09-02T07:27:10.738 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2021-09-02T07:27:10.738 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000216459 s, 2.4 MB/s 2021-09-02T07:27:10.739 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-09-02T07:27:10.793 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-09-02T07:27:10.794 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-02T07:27:10.794 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 6h/6d Inode: 110489 Links: 1 2021-09-02T07:27:10.794 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-02T07:27:10.795 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2021-09-02T07:27:10.795 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2021-09-02 07:27:03.980674326 +0000 2021-09-02T07:27:10.795 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2021-09-02 07:24:07.089519649 +0000 2021-09-02T07:27:10.795 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2021-09-02 07:24:07.089519649 +0000 2021-09-02T07:27:10.795 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: - 2021-09-02T07:27:10.803 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2021-09-02T07:27:10.803 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2021-09-02T07:27:10.803 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000215062 s, 2.4 MB/s 2021-09-02T07:27:10.805 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-09-02T07:27:10.861 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-09-02T07:27:10.861 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-02T07:27:10.862 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 6h/6d Inode: 106233 Links: 1 2021-09-02T07:27:10.862 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-02T07:27:10.862 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2021-09-02T07:27:10.862 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2021-09-02 07:27:03.980674326 +0000 2021-09-02T07:27:10.863 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2021-09-02 07:24:07.484515387 +0000 2021-09-02T07:27:10.863 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2021-09-02 07:24:07.484515387 +0000 2021-09-02T07:27:10.863 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: - 2021-09-02T07:27:10.870 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2021-09-02T07:27:10.871 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2021-09-02T07:27:10.871 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000210933 s, 2.4 MB/s 2021-09-02T07:27:10.873 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-09-02T07:27:10.927 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-09-02T07:27:10.927 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-02T07:27:10.928 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 6h/6d Inode: 107097 Links: 1 2021-09-02T07:27:10.928 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-02T07:27:10.928 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2021-09-02T07:27:10.928 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2021-09-02 07:27:03.980674326 +0000 2021-09-02T07:27:10.929 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2021-09-02 07:24:07.849511447 +0000 2021-09-02T07:27:10.929 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2021-09-02 07:24:07.849511447 +0000 2021-09-02T07:27:10.929 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: - 2021-09-02T07:27:10.937 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2021-09-02T07:27:10.937 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2021-09-02T07:27:10.938 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000205841 s, 2.5 MB/s 2021-09-02T07:27:10.940 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-02T07:27:10.940 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi175.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-02T07:27:10.941 INFO:tasks.ceph:Generating config... 2021-09-02T07:27:10.943 INFO:tasks.ceph:[client] debug rgw = 20 2021-09-02T07:27:10.944 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-09-02T07:27:10.944 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-09-02T07:27:10.945 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-02T07:27:10.946 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-09-02T07:27:10.946 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-09-02T07:27:10.947 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-09-02T07:27:10.947 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-09-02T07:27:10.948 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2021-09-02T07:27:10.948 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-09-02T07:27:10.949 INFO:tasks.ceph:[client] setgroup = ceph 2021-09-02T07:27:10.950 INFO:tasks.ceph:[client] setuser = ceph 2021-09-02T07:27:10.950 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-09-02T07:27:10.951 INFO:tasks.ceph:[mgr] debug ms = 1 2021-09-02T07:27:10.951 INFO:tasks.ceph:[mon] debug mon = 20 2021-09-02T07:27:10.952 INFO:tasks.ceph:[mon] debug ms = 1 2021-09-02T07:27:10.952 INFO:tasks.ceph:[mon] debug paxos = 20 2021-09-02T07:27:10.953 INFO:tasks.ceph:[osd] debug filestore = 20 2021-09-02T07:27:10.953 INFO:tasks.ceph:[osd] debug journal = 20 2021-09-02T07:27:10.954 INFO:tasks.ceph:[osd] debug ms = 20 2021-09-02T07:27:10.955 INFO:tasks.ceph:[osd] debug osd = 25 2021-09-02T07:27:10.955 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-09-02T07:27:10.956 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-09-02T07:27:10.956 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-09-02T07:27:10.957 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-09-02T07:27:10.958 INFO:tasks.ceph:Setting up c1.mon.a... 2021-09-02T07:27:10.958 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2021-09-02T07:27:11.047 INFO:teuthology.orchestra.run.smithi175.stdout:creating /etc/ceph/c1.keyring 2021-09-02T07:27:11.049 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2021-09-02T07:27:11.087 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-09-02T07:27:11.152 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.175')] 2021-09-02T07:27:11.153 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.175'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2021-09-02T07:27:11.158 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:27:11.158 DEBUG:teuthology.orchestra.run.smithi175:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-09-02T07:27:11.207 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.175 --print /home/ubuntu/cephtest/c1.monmap 2021-09-02T07:27:11.282 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:27:11.282 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:27:11.283 INFO:teuthology.orchestra.run.smithi175.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2021-09-02T07:27:11.284 INFO:teuthology.orchestra.run.smithi175.stdout:monmaptool: generated fsid b7a4a901-53ba-461e-8fb5-de971f32ab82 2021-09-02T07:27:11.284 INFO:teuthology.orchestra.run.smithi175.stdout:setting min_mon_release = octopus 2021-09-02T07:27:11.284 INFO:teuthology.orchestra.run.smithi175.stdout:epoch 0 2021-09-02T07:27:11.285 INFO:teuthology.orchestra.run.smithi175.stdout:fsid b7a4a901-53ba-461e-8fb5-de971f32ab82 2021-09-02T07:27:11.285 INFO:teuthology.orchestra.run.smithi175.stdout:last_changed 2021-09-02T07:27:11.285335+0000 2021-09-02T07:27:11.285 INFO:teuthology.orchestra.run.smithi175.stdout:created 2021-09-02T07:27:11.285335+0000 2021-09-02T07:27:11.285 INFO:teuthology.orchestra.run.smithi175.stdout:min_mon_release 15 (octopus) 2021-09-02T07:27:11.285 INFO:teuthology.orchestra.run.smithi175.stdout:election_strategy: 1 2021-09-02T07:27:11.286 INFO:teuthology.orchestra.run.smithi175.stdout:0: [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] mon.a 2021-09-02T07:27:11.286 INFO:teuthology.orchestra.run.smithi175.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2021-09-02T07:27:11.287 DEBUG:teuthology.orchestra.run.smithi175:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-09-02T07:27:11.338 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID b7a4a901-53ba-461e-8fb5-de971f32ab82... 2021-09-02T07:27:11.340 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-09-02T07:27:11.381 DEBUG:teuthology.orchestra.run.smithi197:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-09-02T07:27:11.419 INFO:teuthology.orchestra.run.smithi175.stdout:[global] 2021-09-02T07:27:11.420 INFO:teuthology.orchestra.run.smithi175.stdout: chdir = "" 2021-09-02T07:27:11.420 INFO:teuthology.orchestra.run.smithi175.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-02T07:27:11.420 INFO:teuthology.orchestra.run.smithi175.stdout: auth supported = cephx 2021-09-02T07:27:11.420 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.421 INFO:teuthology.orchestra.run.smithi175.stdout: filestore xattr use omap = true 2021-09-02T07:27:11.421 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.421 INFO:teuthology.orchestra.run.smithi175.stdout: mon clock drift allowed = 1.000 2021-09-02T07:27:11.421 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.421 INFO:teuthology.orchestra.run.smithi175.stdout: osd crush chooseleaf type = 0 2021-09-02T07:27:11.422 INFO:teuthology.orchestra.run.smithi175.stdout: auth debug = true 2021-09-02T07:27:11.422 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.422 INFO:teuthology.orchestra.run.smithi175.stdout: ms die on old message = true 2021-09-02T07:27:11.422 INFO:teuthology.orchestra.run.smithi175.stdout: ms die on bug = true 2021-09-02T07:27:11.423 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.423 INFO:teuthology.orchestra.run.smithi175.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-02T07:27:11.423 INFO:teuthology.orchestra.run.smithi175.stdout: mon pg warn max object skew = 0 2021-09-02T07:27:11.423 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.423 INFO:teuthology.orchestra.run.smithi175.stdout: # disable pg_autoscaler by default for new pools 2021-09-02T07:27:11.424 INFO:teuthology.orchestra.run.smithi175.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-02T07:27:11.424 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.424 INFO:teuthology.orchestra.run.smithi175.stdout: osd pool default size = 2 2021-09-02T07:27:11.424 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.425 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd allow primary affinity = true 2021-09-02T07:27:11.425 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd allow pg remap = true 2021-09-02T07:27:11.425 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on legacy crush tunables = false 2021-09-02T07:27:11.425 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on crush straw calc version zero = false 2021-09-02T07:27:11.425 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on no sortbitwise = false 2021-09-02T07:27:11.426 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on osd down out interval zero = false 2021-09-02T07:27:11.426 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on too few osds = false 2021-09-02T07:27:11.426 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-02T07:27:11.426 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-02T07:27:11.427 INFO:teuthology.orchestra.run.smithi175.stdout: mon_allow_pool_size_one = true 2021-09-02T07:27:11.427 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.427 INFO:teuthology.orchestra.run.smithi175.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-02T07:27:11.427 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.427 INFO:teuthology.orchestra.run.smithi175.stdout: osd default data pool replay window = 5 2021-09-02T07:27:11.428 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.428 INFO:teuthology.orchestra.run.smithi175.stdout: mon allow pool delete = true 2021-09-02T07:27:11.428 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.428 INFO:teuthology.orchestra.run.smithi175.stdout: mon cluster log file level = debug 2021-09-02T07:27:11.429 INFO:teuthology.orchestra.run.smithi175.stdout: debug asserts on shutdown = true 2021-09-02T07:27:11.429 INFO:teuthology.orchestra.run.smithi175.stdout: mon health detail to clog = false 2021-09-02T07:27:11.429 INFO:teuthology.orchestra.run.smithi175.stdout: mon host = 172.21.15.175 2021-09-02T07:27:11.429 INFO:teuthology.orchestra.run.smithi175.stdout: fsid = b7a4a901-53ba-461e-8fb5-de971f32ab82 2021-09-02T07:27:11.429 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.430 INFO:teuthology.orchestra.run.smithi175.stdout:[osd] 2021-09-02T07:27:11.430 INFO:teuthology.orchestra.run.smithi175.stdout: osd journal size = 100 2021-09-02T07:27:11.430 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.430 INFO:teuthology.orchestra.run.smithi175.stdout: osd scrub load threshold = 5.0 2021-09-02T07:27:11.431 INFO:teuthology.orchestra.run.smithi175.stdout: osd scrub max interval = 600 2021-09-02T07:27:11.431 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.431 INFO:teuthology.orchestra.run.smithi175.stdout: osd recover clone overlap = true 2021-09-02T07:27:11.431 INFO:teuthology.orchestra.run.smithi175.stdout: osd recovery max chunk = 1048576 2021-09-02T07:27:11.431 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.432 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug shutdown = true 2021-09-02T07:27:11.432 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug op order = true 2021-09-02T07:27:11.432 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug verify stray on activate = true 2021-09-02T07:27:11.432 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.433 INFO:teuthology.orchestra.run.smithi175.stdout: osd open classes on start = true 2021-09-02T07:27:11.433 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug pg log writeout = true 2021-09-02T07:27:11.433 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.433 INFO:teuthology.orchestra.run.smithi175.stdout: osd deep scrub update digest min age = 30 2021-09-02T07:27:11.433 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.434 INFO:teuthology.orchestra.run.smithi175.stdout: osd map max advance = 10 2021-09-02T07:27:11.434 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.434 INFO:teuthology.orchestra.run.smithi175.stdout: journal zero on create = true 2021-09-02T07:27:11.434 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.435 INFO:teuthology.orchestra.run.smithi175.stdout: filestore ondisk finisher threads = 3 2021-09-02T07:27:11.435 INFO:teuthology.orchestra.run.smithi175.stdout: filestore apply finisher threads = 3 2021-09-02T07:27:11.435 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.435 INFO:teuthology.orchestra.run.smithi175.stdout: bdev debug aio = true 2021-09-02T07:27:11.435 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug misdirected ops = true 2021-09-02T07:27:11.436 INFO:teuthology.orchestra.run.smithi175.stdout: debug filestore = 20 2021-09-02T07:27:11.436 INFO:teuthology.orchestra.run.smithi175.stdout: debug journal = 20 2021-09-02T07:27:11.436 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 20 2021-09-02T07:27:11.436 INFO:teuthology.orchestra.run.smithi175.stdout: debug osd = 25 2021-09-02T07:27:11.436 INFO:teuthology.orchestra.run.smithi175.stdout: rocksdb delete range threshold = 0 2021-09-02T07:27:11.437 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.437 INFO:teuthology.orchestra.run.smithi175.stdout:[mgr] 2021-09-02T07:27:11.437 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2021-09-02T07:27:11.437 INFO:teuthology.orchestra.run.smithi175.stdout: debug mgr = 20 2021-09-02T07:27:11.438 INFO:teuthology.orchestra.run.smithi175.stdout: debug mon = 20 2021-09-02T07:27:11.438 INFO:teuthology.orchestra.run.smithi175.stdout: debug auth = 20 2021-09-02T07:27:11.438 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min pgs per osd = 4 2021-09-02T07:27:11.438 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min bytes per osd = 10 2021-09-02T07:27:11.438 INFO:teuthology.orchestra.run.smithi175.stdout: mgr/telemetry/nag = false 2021-09-02T07:27:11.439 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.439 INFO:teuthology.orchestra.run.smithi175.stdout:[mon] 2021-09-02T07:27:11.439 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2021-09-02T07:27:11.439 INFO:teuthology.orchestra.run.smithi175.stdout: debug mon = 20 2021-09-02T07:27:11.440 INFO:teuthology.orchestra.run.smithi175.stdout: debug paxos = 20 2021-09-02T07:27:11.440 INFO:teuthology.orchestra.run.smithi175.stdout: debug auth = 20 2021-09-02T07:27:11.440 INFO:teuthology.orchestra.run.smithi175.stdout: mon data avail warn = 5 2021-09-02T07:27:11.440 INFO:teuthology.orchestra.run.smithi175.stdout: mon mgr mkfs grace = 240 2021-09-02T07:27:11.441 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min pgs per osd = 4 2021-09-02T07:27:11.441 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd reporter subtree level = osd 2021-09-02T07:27:11.441 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd prime pg temp = true 2021-09-02T07:27:11.441 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min bytes per osd = 10 2021-09-02T07:27:11.442 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.442 INFO:teuthology.orchestra.run.smithi175.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-02T07:27:11.442 INFO:teuthology.orchestra.run.smithi175.stdout: auth mon ticket ttl = 660 # 11m 2021-09-02T07:27:11.442 INFO:teuthology.orchestra.run.smithi175.stdout: auth service ticket ttl = 240 # 4m 2021-09-02T07:27:11.442 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.443 INFO:teuthology.orchestra.run.smithi175.stdout: # don't complain about insecure global_id in the test suite 2021-09-02T07:27:11.443 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-02T07:27:11.443 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-02T07:27:11.443 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:11.444 INFO:teuthology.orchestra.run.smithi175.stdout:[client] 2021-09-02T07:27:11.444 INFO:teuthology.orchestra.run.smithi175.stdout: rgw cache enabled = true 2021-09-02T07:27:11.444 INFO:teuthology.orchestra.run.smithi175.stdout: rgw enable ops log = true 2021-09-02T07:27:11.444 INFO:teuthology.orchestra.run.smithi175.stdout: rgw enable usage log = true 2021-09-02T07:27:11.444 INFO:teuthology.orchestra.run.smithi175.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-02T07:27:11.445 INFO:teuthology.orchestra.run.smithi175.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-02T07:27:11.445 INFO:teuthology.orchestra.run.smithi175.stdout: debug rgw = 20 2021-09-02T07:27:11.445 INFO:teuthology.orchestra.run.smithi175.stdout: rgw crypt require ssl = False 2021-09-02T07:27:11.445 INFO:teuthology.orchestra.run.smithi175.stdout: rgw crypt s3 kms backend = testing 2021-09-02T07:27:11.445 INFO:teuthology.orchestra.run.smithi175.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-02T07:27:11.446 INFO:teuthology.orchestra.run.smithi175.stdout: rgw curl low speed time = 300 2021-09-02T07:27:11.446 INFO:teuthology.orchestra.run.smithi175.stdout: rgw data log num shards = 4 2021-09-02T07:27:11.446 INFO:teuthology.orchestra.run.smithi175.stdout: rgw md log max shards = 4 2021-09-02T07:27:11.446 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync log trim interval = 0 2021-09-02T07:27:11.447 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync meta inject err probability = 0.1 2021-09-02T07:27:11.447 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync obj etag verify = True 2021-09-02T07:27:11.447 INFO:teuthology.orchestra.run.smithi175.stdout: setgroup = ceph 2021-09-02T07:27:11.447 INFO:teuthology.orchestra.run.smithi175.stdout: setuser = ceph 2021-09-02T07:27:11.447 INFO:teuthology.orchestra.run.smithi175.stdout:[mon.a] 2021-09-02T07:27:11.448 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.0] 2021-09-02T07:27:11.448 INFO:teuthology.orchestra.run.smithi175.stdout: osd_max_omap_entries_per_request = 10 2021-09-02T07:27:11.448 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.1] 2021-09-02T07:27:11.448 INFO:teuthology.orchestra.run.smithi175.stdout: osd_max_omap_entries_per_request = 1000 2021-09-02T07:27:11.448 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.2] 2021-09-02T07:27:11.449 INFO:teuthology.orchestra.run.smithi175.stdout: osd_max_omap_entries_per_request = 10000 2021-09-02T07:27:11.450 INFO:teuthology.orchestra.run.smithi197.stdout:[global] 2021-09-02T07:27:11.451 INFO:teuthology.orchestra.run.smithi197.stdout: chdir = "" 2021-09-02T07:27:11.451 INFO:teuthology.orchestra.run.smithi197.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-02T07:27:11.451 INFO:teuthology.orchestra.run.smithi197.stdout: auth supported = cephx 2021-09-02T07:27:11.451 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.452 INFO:teuthology.orchestra.run.smithi197.stdout: filestore xattr use omap = true 2021-09-02T07:27:11.452 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.452 INFO:teuthology.orchestra.run.smithi197.stdout: mon clock drift allowed = 1.000 2021-09-02T07:27:11.452 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.453 INFO:teuthology.orchestra.run.smithi197.stdout: osd crush chooseleaf type = 0 2021-09-02T07:27:11.453 INFO:teuthology.orchestra.run.smithi197.stdout: auth debug = true 2021-09-02T07:27:11.453 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.453 INFO:teuthology.orchestra.run.smithi197.stdout: ms die on old message = true 2021-09-02T07:27:11.453 INFO:teuthology.orchestra.run.smithi197.stdout: ms die on bug = true 2021-09-02T07:27:11.454 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.454 INFO:teuthology.orchestra.run.smithi197.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-02T07:27:11.454 INFO:teuthology.orchestra.run.smithi197.stdout: mon pg warn max object skew = 0 2021-09-02T07:27:11.454 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.455 INFO:teuthology.orchestra.run.smithi197.stdout: # disable pg_autoscaler by default for new pools 2021-09-02T07:27:11.455 INFO:teuthology.orchestra.run.smithi197.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-02T07:27:11.455 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.455 INFO:teuthology.orchestra.run.smithi197.stdout: osd pool default size = 2 2021-09-02T07:27:11.455 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.456 INFO:teuthology.orchestra.run.smithi197.stdout: mon osd allow primary affinity = true 2021-09-02T07:27:11.456 INFO:teuthology.orchestra.run.smithi197.stdout: mon osd allow pg remap = true 2021-09-02T07:27:11.456 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on legacy crush tunables = false 2021-09-02T07:27:11.456 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on crush straw calc version zero = false 2021-09-02T07:27:11.456 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on no sortbitwise = false 2021-09-02T07:27:11.457 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on osd down out interval zero = false 2021-09-02T07:27:11.457 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on too few osds = false 2021-09-02T07:27:11.457 INFO:teuthology.orchestra.run.smithi197.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-02T07:27:11.457 INFO:teuthology.orchestra.run.smithi197.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-02T07:27:11.457 INFO:teuthology.orchestra.run.smithi197.stdout: mon_allow_pool_size_one = true 2021-09-02T07:27:11.458 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.458 INFO:teuthology.orchestra.run.smithi197.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-02T07:27:11.458 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.458 INFO:teuthology.orchestra.run.smithi197.stdout: osd default data pool replay window = 5 2021-09-02T07:27:11.459 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.459 INFO:teuthology.orchestra.run.smithi197.stdout: mon allow pool delete = true 2021-09-02T07:27:11.459 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.459 INFO:teuthology.orchestra.run.smithi197.stdout: mon cluster log file level = debug 2021-09-02T07:27:11.459 INFO:teuthology.orchestra.run.smithi197.stdout: debug asserts on shutdown = true 2021-09-02T07:27:11.460 INFO:teuthology.orchestra.run.smithi197.stdout: mon health detail to clog = false 2021-09-02T07:27:11.460 INFO:teuthology.orchestra.run.smithi197.stdout: mon host = 172.21.15.175 2021-09-02T07:27:11.460 INFO:teuthology.orchestra.run.smithi197.stdout: fsid = b7a4a901-53ba-461e-8fb5-de971f32ab82 2021-09-02T07:27:11.460 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.460 INFO:teuthology.orchestra.run.smithi197.stdout:[osd] 2021-09-02T07:27:11.461 INFO:teuthology.orchestra.run.smithi197.stdout: osd journal size = 100 2021-09-02T07:27:11.461 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.461 INFO:teuthology.orchestra.run.smithi197.stdout: osd scrub load threshold = 5.0 2021-09-02T07:27:11.461 INFO:teuthology.orchestra.run.smithi197.stdout: osd scrub max interval = 600 2021-09-02T07:27:11.461 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.462 INFO:teuthology.orchestra.run.smithi197.stdout: osd recover clone overlap = true 2021-09-02T07:27:11.462 INFO:teuthology.orchestra.run.smithi197.stdout: osd recovery max chunk = 1048576 2021-09-02T07:27:11.462 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.462 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug shutdown = true 2021-09-02T07:27:11.463 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug op order = true 2021-09-02T07:27:11.463 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug verify stray on activate = true 2021-09-02T07:27:11.463 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.463 INFO:teuthology.orchestra.run.smithi197.stdout: osd open classes on start = true 2021-09-02T07:27:11.463 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug pg log writeout = true 2021-09-02T07:27:11.464 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.464 INFO:teuthology.orchestra.run.smithi197.stdout: osd deep scrub update digest min age = 30 2021-09-02T07:27:11.464 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.464 INFO:teuthology.orchestra.run.smithi197.stdout: osd map max advance = 10 2021-09-02T07:27:11.465 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.465 INFO:teuthology.orchestra.run.smithi197.stdout: journal zero on create = true 2021-09-02T07:27:11.465 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.465 INFO:teuthology.orchestra.run.smithi197.stdout: filestore ondisk finisher threads = 3 2021-09-02T07:27:11.465 INFO:teuthology.orchestra.run.smithi197.stdout: filestore apply finisher threads = 3 2021-09-02T07:27:11.466 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.466 INFO:teuthology.orchestra.run.smithi197.stdout: bdev debug aio = true 2021-09-02T07:27:11.466 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug misdirected ops = true 2021-09-02T07:27:11.466 INFO:teuthology.orchestra.run.smithi197.stdout: debug filestore = 20 2021-09-02T07:27:11.466 INFO:teuthology.orchestra.run.smithi197.stdout: debug journal = 20 2021-09-02T07:27:11.467 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 20 2021-09-02T07:27:11.467 INFO:teuthology.orchestra.run.smithi197.stdout: debug osd = 25 2021-09-02T07:27:11.467 INFO:teuthology.orchestra.run.smithi197.stdout: rocksdb delete range threshold = 0 2021-09-02T07:27:11.467 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.468 INFO:teuthology.orchestra.run.smithi197.stdout:[mgr] 2021-09-02T07:27:11.468 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 1 2021-09-02T07:27:11.468 INFO:teuthology.orchestra.run.smithi197.stdout: debug mgr = 20 2021-09-02T07:27:11.468 INFO:teuthology.orchestra.run.smithi197.stdout: debug mon = 20 2021-09-02T07:27:11.468 INFO:teuthology.orchestra.run.smithi197.stdout: debug auth = 20 2021-09-02T07:27:11.469 INFO:teuthology.orchestra.run.smithi197.stdout: mon reweight min pgs per osd = 4 2021-09-02T07:27:11.469 INFO:teuthology.orchestra.run.smithi197.stdout: mon reweight min bytes per osd = 10 2021-09-02T07:27:11.469 INFO:teuthology.orchestra.run.smithi197.stdout: mgr/telemetry/nag = false 2021-09-02T07:27:11.469 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.469 INFO:teuthology.orchestra.run.smithi197.stdout:[mon] 2021-09-02T07:27:11.470 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 1 2021-09-02T07:27:11.470 INFO:teuthology.orchestra.run.smithi197.stdout: debug mon = 20 2021-09-02T07:27:11.470 INFO:teuthology.orchestra.run.smithi197.stdout: debug paxos = 20 2021-09-02T07:27:11.470 INFO:teuthology.orchestra.run.smithi197.stdout: debug auth = 20 2021-09-02T07:27:11.471 INFO:teuthology.orchestra.run.smithi197.stdout: mon data avail warn = 5 2021-09-02T07:27:11.471 INFO:teuthology.orchestra.run.smithi197.stdout: mon mgr mkfs grace = 240 2021-09-02T07:27:11.471 INFO:teuthology.orchestra.run.smithi197.stdout: mon reweight min pgs per osd = 4 2021-09-02T07:27:11.471 INFO:teuthology.orchestra.run.smithi197.stdout: mon osd reporter subtree level = osd 2021-09-02T07:27:11.471 INFO:teuthology.orchestra.run.smithi197.stdout: mon osd prime pg temp = true 2021-09-02T07:27:11.472 INFO:teuthology.orchestra.run.smithi197.stdout: mon reweight min bytes per osd = 10 2021-09-02T07:27:11.472 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.472 INFO:teuthology.orchestra.run.smithi197.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-02T07:27:11.472 INFO:teuthology.orchestra.run.smithi197.stdout: auth mon ticket ttl = 660 # 11m 2021-09-02T07:27:11.472 INFO:teuthology.orchestra.run.smithi197.stdout: auth service ticket ttl = 240 # 4m 2021-09-02T07:27:11.473 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.473 INFO:teuthology.orchestra.run.smithi197.stdout: # don't complain about insecure global_id in the test suite 2021-09-02T07:27:11.473 INFO:teuthology.orchestra.run.smithi197.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-02T07:27:11.473 INFO:teuthology.orchestra.run.smithi197.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-02T07:27:11.474 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:11.474 INFO:teuthology.orchestra.run.smithi197.stdout:[client] 2021-09-02T07:27:11.474 INFO:teuthology.orchestra.run.smithi197.stdout: rgw cache enabled = true 2021-09-02T07:27:11.474 INFO:teuthology.orchestra.run.smithi197.stdout: rgw enable ops log = true 2021-09-02T07:27:11.474 INFO:teuthology.orchestra.run.smithi197.stdout: rgw enable usage log = true 2021-09-02T07:27:11.475 INFO:teuthology.orchestra.run.smithi197.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-02T07:27:11.475 INFO:teuthology.orchestra.run.smithi197.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-02T07:27:11.475 INFO:teuthology.orchestra.run.smithi197.stdout: debug rgw = 20 2021-09-02T07:27:11.475 INFO:teuthology.orchestra.run.smithi197.stdout: rgw crypt require ssl = False 2021-09-02T07:27:11.475 INFO:teuthology.orchestra.run.smithi197.stdout: rgw crypt s3 kms backend = testing 2021-09-02T07:27:11.476 INFO:teuthology.orchestra.run.smithi197.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-02T07:27:11.476 INFO:teuthology.orchestra.run.smithi197.stdout: rgw curl low speed time = 300 2021-09-02T07:27:11.476 INFO:teuthology.orchestra.run.smithi197.stdout: rgw data log num shards = 4 2021-09-02T07:27:11.476 INFO:teuthology.orchestra.run.smithi197.stdout: rgw md log max shards = 4 2021-09-02T07:27:11.477 INFO:teuthology.orchestra.run.smithi197.stdout: rgw sync log trim interval = 0 2021-09-02T07:27:11.477 INFO:teuthology.orchestra.run.smithi197.stdout: rgw sync meta inject err probability = 0.1 2021-09-02T07:27:11.477 INFO:teuthology.orchestra.run.smithi197.stdout: rgw sync obj etag verify = True 2021-09-02T07:27:11.477 INFO:teuthology.orchestra.run.smithi197.stdout: setgroup = ceph 2021-09-02T07:27:11.477 INFO:teuthology.orchestra.run.smithi197.stdout: setuser = ceph 2021-09-02T07:27:11.478 INFO:teuthology.orchestra.run.smithi197.stdout:[mon.a] 2021-09-02T07:27:11.478 INFO:teuthology.orchestra.run.smithi197.stdout:[osd.0] 2021-09-02T07:27:11.478 INFO:teuthology.orchestra.run.smithi197.stdout: osd_max_omap_entries_per_request = 10 2021-09-02T07:27:11.478 INFO:teuthology.orchestra.run.smithi197.stdout:[osd.1] 2021-09-02T07:27:11.478 INFO:teuthology.orchestra.run.smithi197.stdout: osd_max_omap_entries_per_request = 1000 2021-09-02T07:27:11.479 INFO:teuthology.orchestra.run.smithi197.stdout:[osd.2] 2021-09-02T07:27:11.479 INFO:teuthology.orchestra.run.smithi197.stdout: osd_max_omap_entries_per_request = 10000 2021-09-02T07:27:11.479 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2021-09-02T07:27:11.480 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2021-09-02T07:27:11.541 INFO:tasks.ceph:Copying monmap to all nodes... 2021-09-02T07:27:11.542 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:27:11.542 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2021-09-02T07:27:11.557 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:27:11.557 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2021-09-02T07:27:11.611 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi175.front.sepia.ceph.com 2021-09-02T07:27:11.612 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:27:11.612 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/ceph/c1.keyring 2021-09-02T07:27:11.612 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-09-02T07:27:11.684 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:27:11.685 DEBUG:teuthology.orchestra.run.smithi175:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-09-02T07:27:11.741 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi197.front.sepia.ceph.com 2021-09-02T07:27:11.742 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:27:11.743 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd of=/etc/ceph/c1.keyring 2021-09-02T07:27:11.743 DEBUG:teuthology.orchestra.run.smithi197:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-09-02T07:27:11.779 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:27:11.780 DEBUG:teuthology.orchestra.run.smithi197:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-09-02T07:27:11.834 INFO:tasks.ceph:Setting up mon nodes... 2021-09-02T07:27:11.835 INFO:tasks.ceph:Setting up mgr nodes... 2021-09-02T07:27:11.835 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2021-09-02T07:27:11.882 INFO:teuthology.orchestra.run.smithi175.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2021-09-02T07:27:11.884 INFO:tasks.ceph:Setting up mds nodes... 2021-09-02T07:27:11.884 INFO:tasks.ceph_client:Setting up client nodes... 2021-09-02T07:27:11.885 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2021-09-02T07:27:11.922 INFO:teuthology.orchestra.run.smithi175.stdout:creating /etc/ceph/c1.client.0.keyring 2021-09-02T07:27:11.934 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2021-09-02T07:27:12.011 INFO:teuthology.orchestra.run.smithi175.stdout:creating /etc/ceph/c1.client.1.keyring 2021-09-02T07:27:12.023 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-09-02T07:27:12.023 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi175.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-02T07:27:12.024 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2021-09-02T07:27:12.086 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-02T07:27:12.087 INFO:tasks.ceph:role: c1.osd.0 2021-09-02T07:27:12.087 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi175.front.sepia.ceph.com 2021-09-02T07:27:12.088 DEBUG:teuthology.orchestra.run.smithi175:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-09-02T07:27:12.158 INFO:teuthology.orchestra.run.smithi175.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-09-02T07:27:12.158 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-02T07:27:12.159 INFO:teuthology.orchestra.run.smithi175.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-02T07:27:12.159 INFO:teuthology.orchestra.run.smithi175.stdout: = reflink=1 2021-09-02T07:27:12.159 INFO:teuthology.orchestra.run.smithi175.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-02T07:27:12.159 INFO:teuthology.orchestra.run.smithi175.stdout: = sunit=0 swidth=0 blks 2021-09-02T07:27:12.160 INFO:teuthology.orchestra.run.smithi175.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-02T07:27:12.160 INFO:teuthology.orchestra.run.smithi175.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-02T07:27:12.160 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-02T07:27:12.160 INFO:teuthology.orchestra.run.smithi175.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-02T07:27:12.519 INFO:teuthology.orchestra.run.smithi175.stdout:Discarding blocks...Done. 2021-09-02T07:27:12.520 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi175.front.sepia.ceph.com -o noatime 2021-09-02T07:27:12.521 DEBUG:teuthology.orchestra.run.smithi175:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2021-09-02T07:27:12.652 DEBUG:teuthology.orchestra.run.smithi175:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2021-09-02T07:27:12.681 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2021-09-02T07:27:12.745 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-02T07:27:12.745 INFO:tasks.ceph:role: c1.osd.1 2021-09-02T07:27:12.746 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi175.front.sepia.ceph.com 2021-09-02T07:27:12.746 DEBUG:teuthology.orchestra.run.smithi175:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-09-02T07:27:12.816 INFO:teuthology.orchestra.run.smithi175.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-09-02T07:27:12.816 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-02T07:27:12.816 INFO:teuthology.orchestra.run.smithi175.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-02T07:27:12.817 INFO:teuthology.orchestra.run.smithi175.stdout: = reflink=1 2021-09-02T07:27:12.817 INFO:teuthology.orchestra.run.smithi175.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-02T07:27:12.817 INFO:teuthology.orchestra.run.smithi175.stdout: = sunit=0 swidth=0 blks 2021-09-02T07:27:12.817 INFO:teuthology.orchestra.run.smithi175.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-02T07:27:12.817 INFO:teuthology.orchestra.run.smithi175.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-02T07:27:12.818 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-02T07:27:12.818 INFO:teuthology.orchestra.run.smithi175.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-02T07:27:13.176 INFO:teuthology.orchestra.run.smithi175.stdout:Discarding blocks...Done. 2021-09-02T07:27:13.177 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi175.front.sepia.ceph.com -o noatime 2021-09-02T07:27:13.178 DEBUG:teuthology.orchestra.run.smithi175:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2021-09-02T07:27:13.265 DEBUG:teuthology.orchestra.run.smithi175:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2021-09-02T07:27:13.293 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2021-09-02T07:27:13.357 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-02T07:27:13.357 INFO:tasks.ceph:role: c1.osd.2 2021-09-02T07:27:13.357 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi175.front.sepia.ceph.com 2021-09-02T07:27:13.358 DEBUG:teuthology.orchestra.run.smithi175:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-09-02T07:27:13.432 INFO:teuthology.orchestra.run.smithi175.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-09-02T07:27:13.701 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-02T07:27:13.702 INFO:teuthology.orchestra.run.smithi175.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-02T07:27:13.702 INFO:teuthology.orchestra.run.smithi175.stdout: = reflink=1 2021-09-02T07:27:13.702 INFO:teuthology.orchestra.run.smithi175.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-02T07:27:13.702 INFO:teuthology.orchestra.run.smithi175.stdout: = sunit=0 swidth=0 blks 2021-09-02T07:27:13.703 INFO:teuthology.orchestra.run.smithi175.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-02T07:27:13.703 INFO:teuthology.orchestra.run.smithi175.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-02T07:27:13.703 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-02T07:27:13.704 INFO:teuthology.orchestra.run.smithi175.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-02T07:27:13.801 INFO:teuthology.orchestra.run.smithi175.stdout:Discarding blocks...Done. 2021-09-02T07:27:13.802 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi175.front.sepia.ceph.com -o noatime 2021-09-02T07:27:13.803 DEBUG:teuthology.orchestra.run.smithi175:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2021-09-02T07:27:13.911 DEBUG:teuthology.orchestra.run.smithi175:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2021-09-02T07:27:13.950 DEBUG:teuthology.orchestra.run.smithi175:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2021-09-02T07:27:14.044 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:27:14.046+0000 7f9bc33680c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2021-09-02T07:27:14.045 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:27:14.047+0000 7f9bc33680c0 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2021-09-02T07:27:14.046 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:27:14.047+0000 7f9bc33680c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-09-02T07:27:14.046 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:27:14.047+0000 7f9bc33680c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-09-02T07:27:14.046 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:27:14.047+0000 7f9bc33680c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-09-02T07:27:14.046 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:27:14.047+0000 7f9bc33680c0 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2021-09-02T07:27:16.130 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2021-09-02T07:27:16.200 DEBUG:teuthology.orchestra.run.smithi175:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2021-09-02T07:27:16.285 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:27:16.286+0000 7fe9e1eb90c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2021-09-02T07:27:16.285 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:27:16.286+0000 7fe9e1eb90c0 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2021-09-02T07:27:16.285 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:27:16.286+0000 7fe9e1eb90c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-09-02T07:27:16.286 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:27:16.286+0000 7fe9e1eb90c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-09-02T07:27:16.286 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:27:16.286+0000 7fe9e1eb90c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-09-02T07:27:16.286 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:27:16.286+0000 7fe9e1eb90c0 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2021-09-02T07:27:18.375 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2021-09-02T07:27:18.404 DEBUG:teuthology.orchestra.run.smithi175:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2021-09-02T07:27:18.490 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:27:18.491+0000 7fe50b5280c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2021-09-02T07:27:18.491 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:27:18.491+0000 7fe50b5280c0 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2021-09-02T07:27:18.491 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:27:18.491+0000 7fe50b5280c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-09-02T07:27:18.491 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:27:18.491+0000 7fe50b5280c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-09-02T07:27:18.492 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:27:18.491+0000 7fe50b5280c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-09-02T07:27:18.492 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:27:18.491+0000 7fe50b5280c0 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2021-09-02T07:27:20.569 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2021-09-02T07:27:20.599 INFO:tasks.ceph:Reading keys from all nodes... 2021-09-02T07:27:20.599 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:27:20.600 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2021-09-02T07:27:20.666 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:27:20.666 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2021-09-02T07:27:20.731 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:27:20.732 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2021-09-02T07:27:20.797 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:27:20.797 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2021-09-02T07:27:20.868 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:27:20.868 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2021-09-02T07:27:20.923 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:27:20.924 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2021-09-02T07:27:20.977 INFO:tasks.ceph:Adding keys to all mons... 2021-09-02T07:27:20.978 DEBUG:teuthology.orchestra.run.smithi175:> sudo tee -a /etc/ceph/c1.keyring 2021-09-02T07:27:21.040 INFO:teuthology.orchestra.run.smithi175.stdout:[mgr.x] 2021-09-02T07:27:21.041 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQDPfDBhJe7BNBAAPqqye5TZAlyRlljIcXB4Pw== 2021-09-02T07:27:21.042 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.0] 2021-09-02T07:27:21.042 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQDSfDBhhM3XAhAA8nCpUYQSIyat15oZdXll7A== 2021-09-02T07:27:21.042 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.1] 2021-09-02T07:27:21.042 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQDUfDBhLbcgERAAMCHujHHiygSg/jWV8L06sg== 2021-09-02T07:27:21.043 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.2] 2021-09-02T07:27:21.043 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQDWfDBhXt9VHRAAJwVSL8GRExye0A/JNMcTIg== 2021-09-02T07:27:21.043 INFO:teuthology.orchestra.run.smithi175.stdout:[client.0] 2021-09-02T07:27:21.043 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQDPfDBhqjMhNxAAQ34TWwSAU2C374dAzV2aHw== 2021-09-02T07:27:21.044 INFO:teuthology.orchestra.run.smithi175.stdout:[client.1] 2021-09-02T07:27:21.044 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQDQfDBhWnvQABAAtCtkb2v6c0F4dPLFT7Z/Sg== 2021-09-02T07:27:21.045 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-09-02T07:27:21.132 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-09-02T07:27:21.177 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-09-02T07:27:21.223 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-09-02T07:27:21.269 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-02T07:27:21.318 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-02T07:27:21.366 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-09-02T07:27:21.367 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2021-09-02T07:27:21.392 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2021-09-02T07:27:21.487 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2021-09-02T07:27:21.510 DEBUG:teuthology.orchestra.run.smithi175:> rm -- /home/ubuntu/cephtest/c1.monmap 2021-09-02T07:27:21.564 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2021-09-02T07:27:21.564 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2021-09-02T07:27:21.565 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2021-09-02T07:27:21.607 INFO:tasks.ceph.c1.mon.a:Started 2021-09-02T07:27:21.608 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2021-09-02T07:27:21.608 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2021-09-02T07:27:21.608 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2021-09-02T07:27:21.611 INFO:tasks.ceph.c1.mgr.x:Started 2021-09-02T07:27:21.611 INFO:tasks.ceph:Setting crush tunables to default 2021-09-02T07:27:21.612 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster c1 osd crush tunables default 2021-09-02T07:27:21.805 INFO:teuthology.orchestra.run.smithi175.stderr:adjusted tunables profile to default 2021-09-02T07:27:21.816 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2021-09-02T07:27:21.816 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:27:21.817 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2021-09-02T07:27:21.842 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:27:21.842 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2021-09-02T07:27:21.906 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:27:21.906 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2021-09-02T07:27:21.970 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster c1 osd new b1eae4f7-a11c-44f2-83e8-0363aa7d3815 0 2021-09-02T07:27:22.209 INFO:teuthology.orchestra.run.smithi175.stdout:0 2021-09-02T07:27:22.221 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster c1 osd new 17265a55-ead4-4ab5-bc20-f221999fe15d 1 2021-09-02T07:27:22.423 INFO:teuthology.orchestra.run.smithi175.stdout:1 2021-09-02T07:27:22.435 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster c1 osd new a51c81b5-d27b-4019-849e-21dee8342b07 2 2021-09-02T07:27:22.641 INFO:teuthology.orchestra.run.smithi175.stdout:2 2021-09-02T07:27:22.654 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2021-09-02T07:27:22.654 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2021-09-02T07:27:22.657 INFO:tasks.ceph.c1.osd.0:Started 2021-09-02T07:27:22.657 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2021-09-02T07:27:22.658 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2021-09-02T07:27:22.660 INFO:tasks.ceph.c1.osd.1:Started 2021-09-02T07:27:22.660 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2021-09-02T07:27:22.660 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2021-09-02T07:27:22.662 INFO:tasks.ceph.c1.osd.2:Started 2021-09-02T07:27:22.663 INFO:tasks.ceph:Waiting for OSDs to come up 2021-09-02T07:27:22.966 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-09-02T07:27:23.145 INFO:teuthology.misc.health.smithi175.stdout: 2021-09-02T07:27:23.145 INFO:teuthology.misc.health.smithi175.stdout:{"epoch":5,"fsid":"b7a4a901-53ba-461e-8fb5-de971f32ab82","created":"2021-09-02T07:27:21.707072+0000","modified":"2021-09-02T07:27:22.642835+0000","last_up_change":"0.000000","last_in_change":"2021-09-02T07:27:22.642835+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"b1eae4f7-a11c-44f2-83e8-0363aa7d3815","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"17265a55-ead4-4ab5-bc20-f221999fe15d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a51c81b5-d27b-4019-849e-21dee8342b07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-02T07:27:23.156 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-09-02T07:27:23.260 INFO:tasks.ceph.c1.osd.1.smithi175.stderr:2021-09-02T07:27:23.262+0000 7fc2dbf160c0 -1 Falling back to public interface 2021-09-02T07:27:23.268 INFO:tasks.ceph.c1.osd.0.smithi175.stderr:2021-09-02T07:27:23.270+0000 7feb219e00c0 -1 Falling back to public interface 2021-09-02T07:27:23.269 INFO:tasks.ceph.c1.osd.2.smithi175.stderr:2021-09-02T07:27:23.271+0000 7ff6232c80c0 -1 Falling back to public interface 2021-09-02T07:27:25.958 INFO:tasks.ceph.c1.osd.1.smithi175.stderr:2021-09-02T07:27:25.960+0000 7fc2dbf160c0 -1 osd.1 0 log_to_monitors {default=true} 2021-09-02T07:27:25.970 INFO:tasks.ceph.c1.osd.2.smithi175.stderr:2021-09-02T07:27:25.972+0000 7ff6232c80c0 -1 osd.2 0 log_to_monitors {default=true} 2021-09-02T07:27:26.014 INFO:tasks.ceph.c1.osd.0.smithi175.stderr:2021-09-02T07:27:26.016+0000 7feb219e00c0 -1 osd.0 0 log_to_monitors {default=true} 2021-09-02T07:27:28.002 INFO:tasks.ceph.c1.osd.1.smithi175.stderr:2021-09-02T07:27:28.003+0000 7fc2d28d4700 -1 osd.1 0 waiting for initial osdmap 2021-09-02T07:27:28.003 INFO:tasks.ceph.c1.osd.1.smithi175.stderr:2021-09-02T07:27:28.005+0000 7fc2cb267700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-02T07:27:28.223 INFO:tasks.ceph.c1.osd.2.smithi175.stderr:2021-09-02T07:27:28.225+0000 7ff619c86700 -1 osd.2 0 waiting for initial osdmap 2021-09-02T07:27:28.228 INFO:tasks.ceph.c1.osd.2.smithi175.stderr:2021-09-02T07:27:28.229+0000 7ff612619700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-02T07:27:29.083 INFO:tasks.ceph.c1.osd.0.smithi175.stderr:2021-09-02T07:27:29.084+0000 7feb1839e700 -1 osd.0 0 waiting for initial osdmap 2021-09-02T07:27:29.085 INFO:tasks.ceph.c1.osd.0.smithi175.stderr:2021-09-02T07:27:29.086+0000 7feb10d31700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-02T07:27:29.461 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-09-02T07:27:29.739 INFO:teuthology.misc.health.smithi175.stdout: 2021-09-02T07:27:29.740 INFO:teuthology.misc.health.smithi175.stdout:{"epoch":9,"fsid":"b7a4a901-53ba-461e-8fb5-de971f32ab82","created":"2021-09-02T07:27:21.707072+0000","modified":"2021-09-02T07:27:29.223452+0000","last_up_change":"2021-09-02T07:27:29.223452+0000","last_in_change":"2021-09-02T07:27:22.642835+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"b1eae4f7-a11c-44f2-83e8-0363aa7d3815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":30434},{"type":"v1","addr":"172.21.15.175:6809","nonce":30434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":30434},{"type":"v1","addr":"172.21.15.175:6811","nonce":30434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6819","nonce":30434},{"type":"v1","addr":"172.21.15.175:6821","nonce":30434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":30434},{"type":"v1","addr":"172.21.15.175:6816","nonce":30434}]},"public_addr":"172.21.15.175:6809/30434","cluster_addr":"172.21.15.175:6811/30434","heartbeat_back_addr":"172.21.15.175:6821/30434","heartbeat_front_addr":"172.21.15.175:6816/30434","state":["exists","up"]},{"osd":1,"uuid":"17265a55-ead4-4ab5-bc20-f221999fe15d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":30435},{"type":"v1","addr":"172.21.15.175:6801","nonce":30435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":30435},{"type":"v1","addr":"172.21.15.175:6803","nonce":30435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":30435},{"type":"v1","addr":"172.21.15.175:6807","nonce":30435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":30435},{"type":"v1","addr":"172.21.15.175:6805","nonce":30435}]},"public_addr":"172.21.15.175:6801/30435","cluster_addr":"172.21.15.175:6803/30435","heartbeat_back_addr":"172.21.15.175:6807/30435","heartbeat_front_addr":"172.21.15.175:6805/30435","state":["exists","up"]},{"osd":2,"uuid":"a51c81b5-d27b-4019-849e-21dee8342b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":30436},{"type":"v1","addr":"172.21.15.175:6813","nonce":30436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":30436},{"type":"v1","addr":"172.21.15.175:6817","nonce":30436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":30436},{"type":"v1","addr":"172.21.15.175:6823","nonce":30436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":30436},{"type":"v1","addr":"172.21.15.175:6820","nonce":30436}]},"public_addr":"172.21.15.175:6813/30436","cluster_addr":"172.21.15.175:6817/30436","heartbeat_back_addr":"172.21.15.175:6823/30436","heartbeat_front_addr":"172.21.15.175:6820/30436","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-02T07:27:26.998145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-02T07:27:29.751 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-09-02T07:27:29.752 INFO:tasks.ceph:Creating RBD pool 2021-09-02T07:27:29.752 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster c1 osd pool create rbd 8 2021-09-02T07:27:30.634 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'rbd' created 2021-09-02T07:27:30.644 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-09-02T07:27:31.638 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rbd' on pool 'rbd' 2021-09-02T07:27:31.654 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2021-09-02T07:27:31.654 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-09-02T07:27:31.655 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-09-02T07:27:31.972 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:31.972 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":11,"fsid":"b7a4a901-53ba-461e-8fb5-de971f32ab82","created":"2021-09-02T07:27:21.707072+0000","modified":"2021-09-02T07:27:31.628970+0000","last_up_change":"2021-09-02T07:27:29.223452+0000","last_in_change":"2021-09-02T07:27:22.642835+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-02T07:27:30.050695+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-02T07:27:30.611548+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b1eae4f7-a11c-44f2-83e8-0363aa7d3815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":30434},{"type":"v1","addr":"172.21.15.175:6809","nonce":30434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":30434},{"type":"v1","addr":"172.21.15.175:6811","nonce":30434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6819","nonce":30434},{"type":"v1","addr":"172.21.15.175:6821","nonce":30434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":30434},{"type":"v1","addr":"172.21.15.175:6816","nonce":30434}]},"public_addr":"172.21.15.175:6809/30434","cluster_addr":"172.21.15.175:6811/30434","heartbeat_back_addr":"172.21.15.175:6821/30434","heartbeat_front_addr":"172.21.15.175:6816/30434","state":["exists","up"]},{"osd":1,"uuid":"17265a55-ead4-4ab5-bc20-f221999fe15d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":30435},{"type":"v1","addr":"172.21.15.175:6801","nonce":30435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":30435},{"type":"v1","addr":"172.21.15.175:6803","nonce":30435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":30435},{"type":"v1","addr":"172.21.15.175:6807","nonce":30435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":30435},{"type":"v1","addr":"172.21.15.175:6805","nonce":30435}]},"public_addr":"172.21.15.175:6801/30435","cluster_addr":"172.21.15.175:6803/30435","heartbeat_back_addr":"172.21.15.175:6807/30435","heartbeat_front_addr":"172.21.15.175:6805/30435","state":["exists","up"]},{"osd":2,"uuid":"a51c81b5-d27b-4019-849e-21dee8342b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":30436},{"type":"v1","addr":"172.21.15.175:6813","nonce":30436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":30436},{"type":"v1","addr":"172.21.15.175:6817","nonce":30436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":30436},{"type":"v1","addr":"172.21.15.175:6823","nonce":30436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":30436},{"type":"v1","addr":"172.21.15.175:6820","nonce":30436}]},"public_addr":"172.21.15.175:6813/30436","cluster_addr":"172.21.15.175:6817/30436","heartbeat_back_addr":"172.21.15.175:6823/30436","heartbeat_front_addr":"172.21.15.175:6820/30436","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-02T07:27:26.993001+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-02T07:27:26.998145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-02T07:27:26.964485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-02T07:27:31.985 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-09-02T07:27:30.050695+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '11', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2021-09-02T07:27:30.611548+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '11', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2021-09-02T07:27:31.986 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get rbd pg_num 2021-09-02T07:27:32.277 INFO:teuthology.orchestra.run.smithi175.stdout:pg_num: 8 2021-09-02T07:27:32.291 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get .mgr pg_num 2021-09-02T07:27:32.577 INFO:teuthology.orchestra.run.smithi175.stdout:pg_num: 1 2021-09-02T07:27:32.590 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2021-09-02T07:27:32.591 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2021-09-02T07:27:32.591 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2021-09-02T07:27:32.918 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:32.931 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":30116},{"type":"v1","addr":"172.21.15.175:6825","nonce":30116}]},"active_addr":"172.21.15.175:6825/30116","active_change":"2021-09-02T07:27:26.580027+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"autoscale_profile":{"name":"autoscale_profile","type":"str","level":"advanced","flags":1,"default_value":"scale-up","min":"","max":"","enum_allowed":[],"desc":"pg_autoscale profiler","long_desc":"Determines the behavior of the autoscaler algorithm `scale-up` means that it starts out with minmum pgs and scales up when there is pressure, `scale-down` means starts out with full pgs and scales down when there is pressure ","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share perf counter metrics summed across the whole cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.175:0","nonce":105154331}]},{"addrvec":[{"type":"v2","addr":"172.21.15.175:0","nonce":2297115737}]},{"addrvec":[{"type":"v2","addr":"172.21.15.175:0","nonce":2875644927}]},{"addrvec":[{"type":"v2","addr":"172.21.15.175:0","nonce":4180082074}]}]}} 2021-09-02T07:27:32.933 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2021-09-02T07:27:32.934 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2021-09-02T07:27:32.934 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-09-02T07:27:33.215 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:33.216 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":12,"fsid":"b7a4a901-53ba-461e-8fb5-de971f32ab82","created":"2021-09-02T07:27:21.707072+0000","modified":"2021-09-02T07:27:32.633262+0000","last_up_change":"2021-09-02T07:27:29.223452+0000","last_in_change":"2021-09-02T07:27:22.642835+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-02T07:27:30.050695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-02T07:27:30.611548+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b1eae4f7-a11c-44f2-83e8-0363aa7d3815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":30434},{"type":"v1","addr":"172.21.15.175:6809","nonce":30434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":30434},{"type":"v1","addr":"172.21.15.175:6811","nonce":30434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6819","nonce":30434},{"type":"v1","addr":"172.21.15.175:6821","nonce":30434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":30434},{"type":"v1","addr":"172.21.15.175:6816","nonce":30434}]},"public_addr":"172.21.15.175:6809/30434","cluster_addr":"172.21.15.175:6811/30434","heartbeat_back_addr":"172.21.15.175:6821/30434","heartbeat_front_addr":"172.21.15.175:6816/30434","state":["exists","up"]},{"osd":1,"uuid":"17265a55-ead4-4ab5-bc20-f221999fe15d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":30435},{"type":"v1","addr":"172.21.15.175:6801","nonce":30435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":30435},{"type":"v1","addr":"172.21.15.175:6803","nonce":30435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":30435},{"type":"v1","addr":"172.21.15.175:6807","nonce":30435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":30435},{"type":"v1","addr":"172.21.15.175:6805","nonce":30435}]},"public_addr":"172.21.15.175:6801/30435","cluster_addr":"172.21.15.175:6803/30435","heartbeat_back_addr":"172.21.15.175:6807/30435","heartbeat_front_addr":"172.21.15.175:6805/30435","state":["exists","up"]},{"osd":2,"uuid":"a51c81b5-d27b-4019-849e-21dee8342b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":30436},{"type":"v1","addr":"172.21.15.175:6813","nonce":30436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":30436},{"type":"v1","addr":"172.21.15.175:6817","nonce":30436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":30436},{"type":"v1","addr":"172.21.15.175:6823","nonce":30436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":30436},{"type":"v1","addr":"172.21.15.175:6820","nonce":30436}]},"public_addr":"172.21.15.175:6813/30436","cluster_addr":"172.21.15.175:6817/30436","heartbeat_back_addr":"172.21.15.175:6823/30436","heartbeat_front_addr":"172.21.15.175:6820/30436","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-02T07:27:26.993001+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-02T07:27:26.998145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-02T07:27:26.964485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-02T07:27:33.228 INFO:tasks.ceph.ceph_manager.c1:all up! 2021-09-02T07:27:33.229 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-09-02T07:27:33.510 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:33.510 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":12,"fsid":"b7a4a901-53ba-461e-8fb5-de971f32ab82","created":"2021-09-02T07:27:21.707072+0000","modified":"2021-09-02T07:27:32.633262+0000","last_up_change":"2021-09-02T07:27:29.223452+0000","last_in_change":"2021-09-02T07:27:22.642835+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-02T07:27:30.050695+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-02T07:27:30.611548+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b1eae4f7-a11c-44f2-83e8-0363aa7d3815","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":30434},{"type":"v1","addr":"172.21.15.175:6809","nonce":30434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":30434},{"type":"v1","addr":"172.21.15.175:6811","nonce":30434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6819","nonce":30434},{"type":"v1","addr":"172.21.15.175:6821","nonce":30434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":30434},{"type":"v1","addr":"172.21.15.175:6816","nonce":30434}]},"public_addr":"172.21.15.175:6809/30434","cluster_addr":"172.21.15.175:6811/30434","heartbeat_back_addr":"172.21.15.175:6821/30434","heartbeat_front_addr":"172.21.15.175:6816/30434","state":["exists","up"]},{"osd":1,"uuid":"17265a55-ead4-4ab5-bc20-f221999fe15d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":30435},{"type":"v1","addr":"172.21.15.175:6801","nonce":30435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":30435},{"type":"v1","addr":"172.21.15.175:6803","nonce":30435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":30435},{"type":"v1","addr":"172.21.15.175:6807","nonce":30435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":30435},{"type":"v1","addr":"172.21.15.175:6805","nonce":30435}]},"public_addr":"172.21.15.175:6801/30435","cluster_addr":"172.21.15.175:6803/30435","heartbeat_back_addr":"172.21.15.175:6807/30435","heartbeat_front_addr":"172.21.15.175:6805/30435","state":["exists","up"]},{"osd":2,"uuid":"a51c81b5-d27b-4019-849e-21dee8342b07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":30436},{"type":"v1","addr":"172.21.15.175:6813","nonce":30436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6815","nonce":30436},{"type":"v1","addr":"172.21.15.175:6817","nonce":30436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":30436},{"type":"v1","addr":"172.21.15.175:6823","nonce":30436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":30436},{"type":"v1","addr":"172.21.15.175:6820","nonce":30436}]},"public_addr":"172.21.15.175:6813/30436","cluster_addr":"172.21.15.175:6817/30436","heartbeat_back_addr":"172.21.15.175:6823/30436","heartbeat_front_addr":"172.21.15.175:6820/30436","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-02T07:27:26.993001+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-02T07:27:26.998145+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-02T07:27:26.964485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-02T07:27:33.524 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2021-09-02T07:27:33.670 INFO:teuthology.orchestra.run.smithi175.stdout:38654705668 2021-09-02T07:27:33.670 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2021-09-02T07:27:33.814 INFO:teuthology.orchestra.run.smithi175.stdout:34359738372 2021-09-02T07:27:33.815 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2021-09-02T07:27:33.960 INFO:teuthology.orchestra.run.smithi175.stdout:38654705668 2021-09-02T07:27:33.961 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-09-02T07:27:34.248 INFO:teuthology.orchestra.run.smithi175.stdout:38654705667 2021-09-02T07:27:34.260 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705668 got 38654705667 for osd.0 2021-09-02T07:27:35.261 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-09-02T07:27:35.559 INFO:teuthology.orchestra.run.smithi175.stdout:38654705668 2021-09-02T07:27:35.571 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705668 got 38654705668 for osd.0 2021-09-02T07:27:35.572 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2021-09-02T07:27:35.854 INFO:teuthology.orchestra.run.smithi175.stdout:34359738372 2021-09-02T07:27:35.868 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.1 2021-09-02T07:27:35.869 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2021-09-02T07:27:36.150 INFO:teuthology.orchestra.run.smithi175.stdout:38654705668 2021-09-02T07:27:36.163 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705668 got 38654705668 for osd.2 2021-09-02T07:27:36.163 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2021-09-02T07:27:36.164 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-09-02T07:27:36.439 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:36.440 INFO:teuthology.orchestra.run.smithi175.stderr:dumped all 2021-09-02T07:27:36.451 INFO:teuthology.orchestra.run.smithi175.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2021-09-02T07:27:34.584514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24712,"kb_used_data":1456,"kb_used_omap":0,"kb_used_meta":23104,"kb_avail":314548088,"statfs":{"total":322122547200,"available":322097242112,"internally_reserved":0,"allocated":1490944,"data_stored":1258898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23658496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-02T07:27:31.637847+0000","last_change":"2021-09-02T07:27:31.637847+0000","last_active":"2021-09-02T07:27:31.637847+0000","last_peered":"2021-09-02T07:27:31.637847+0000","last_clean":"2021-09-02T07:27:31.637847+0000","last_became_active":"2021-09-02T07:27:31.636151+0000","last_became_peered":"2021-09-02T07:27:31.636151+0000","last_unstale":"2021-09-02T07:27:31.637847+0000","last_undegraded":"2021-09-02T07:27:31.637847+0000","last_fullsized":"2021-09-02T07:27:31.637847+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_clean_scrub_stamp":"2021-09-02T07:27:30.626938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-02T07:27:31.638980+0000","last_change":"2021-09-02T07:27:31.638980+0000","last_active":"2021-09-02T07:27:31.638980+0000","last_peered":"2021-09-02T07:27:31.638980+0000","last_clean":"2021-09-02T07:27:31.638980+0000","last_became_active":"2021-09-02T07:27:31.638687+0000","last_became_peered":"2021-09-02T07:27:31.638687+0000","last_unstale":"2021-09-02T07:27:31.638980+0000","last_undegraded":"2021-09-02T07:27:31.638980+0000","last_fullsized":"2021-09-02T07:27:31.638980+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_clean_scrub_stamp":"2021-09-02T07:27:30.626938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-02T07:27:31.640043+0000","last_change":"2021-09-02T07:27:31.638452+0000","last_active":"2021-09-02T07:27:31.640043+0000","last_peered":"2021-09-02T07:27:31.640043+0000","last_clean":"2021-09-02T07:27:31.640043+0000","last_became_active":"2021-09-02T07:27:31.638158+0000","last_became_peered":"2021-09-02T07:27:31.638158+0000","last_unstale":"2021-09-02T07:27:31.640043+0000","last_undegraded":"2021-09-02T07:27:31.640043+0000","last_fullsized":"2021-09-02T07:27:31.640043+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_clean_scrub_stamp":"2021-09-02T07:27:30.626938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-02T07:27:31.636992+0000","last_change":"2021-09-02T07:27:31.636992+0000","last_active":"2021-09-02T07:27:31.636992+0000","last_peered":"2021-09-02T07:27:31.636992+0000","last_clean":"2021-09-02T07:27:31.636992+0000","last_became_active":"2021-09-02T07:27:31.636712+0000","last_became_peered":"2021-09-02T07:27:31.636712+0000","last_unstale":"2021-09-02T07:27:31.636992+0000","last_undegraded":"2021-09-02T07:27:31.636992+0000","last_fullsized":"2021-09-02T07:27:31.636992+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_clean_scrub_stamp":"2021-09-02T07:27:30.626938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-02T07:27:31.639164+0000","last_change":"2021-09-02T07:27:31.639164+0000","last_active":"2021-09-02T07:27:31.639164+0000","last_peered":"2021-09-02T07:27:31.639164+0000","last_clean":"2021-09-02T07:27:31.639164+0000","last_became_active":"2021-09-02T07:27:31.638862+0000","last_became_peered":"2021-09-02T07:27:31.638862+0000","last_unstale":"2021-09-02T07:27:31.639164+0000","last_undegraded":"2021-09-02T07:27:31.639164+0000","last_fullsized":"2021-09-02T07:27:31.639164+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_clean_scrub_stamp":"2021-09-02T07:27:30.626938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-02T07:27:32.093840+0000","last_change":"2021-09-02T07:27:31.641020+0000","last_active":"2021-09-02T07:27:32.093840+0000","last_peered":"2021-09-02T07:27:32.093840+0000","last_clean":"2021-09-02T07:27:32.093840+0000","last_became_active":"2021-09-02T07:27:31.640723+0000","last_became_peered":"2021-09-02T07:27:31.640723+0000","last_unstale":"2021-09-02T07:27:32.093840+0000","last_undegraded":"2021-09-02T07:27:32.093840+0000","last_fullsized":"2021-09-02T07:27:32.093840+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_clean_scrub_stamp":"2021-09-02T07:27:30.626938+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-02T07:27:31.637997+0000","last_change":"2021-09-02T07:27:31.637997+0000","last_active":"2021-09-02T07:27:31.637997+0000","last_peered":"2021-09-02T07:27:31.637997+0000","last_clean":"2021-09-02T07:27:31.637997+0000","last_became_active":"2021-09-02T07:27:31.637695+0000","last_became_peered":"2021-09-02T07:27:31.637695+0000","last_unstale":"2021-09-02T07:27:31.637997+0000","last_undegraded":"2021-09-02T07:27:31.637997+0000","last_fullsized":"2021-09-02T07:27:31.637997+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_clean_scrub_stamp":"2021-09-02T07:27:30.626938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-02T07:27:31.636089+0000","last_change":"2021-09-02T07:27:31.636089+0000","last_active":"2021-09-02T07:27:31.636089+0000","last_peered":"2021-09-02T07:27:31.636089+0000","last_clean":"2021-09-02T07:27:31.636089+0000","last_became_active":"2021-09-02T07:27:31.635746+0000","last_became_peered":"2021-09-02T07:27:31.635746+0000","last_unstale":"2021-09-02T07:27:31.636089+0000","last_undegraded":"2021-09-02T07:27:31.636089+0000","last_fullsized":"2021-09-02T07:27:31.636089+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_clean_scrub_stamp":"2021-09-02T07:27:30.626938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-02T07:27:31.641873+0000","last_change":"2021-09-02T07:27:31.640453+0000","last_active":"2021-09-02T07:27:31.641873+0000","last_peered":"2021-09-02T07:27:31.641873+0000","last_clean":"2021-09-02T07:27:31.641873+0000","last_became_active":"2021-09-02T07:27:31.639962+0000","last_became_peered":"2021-09-02T07:27:31.639962+0000","last_unstale":"2021-09-02T07:27:31.641873+0000","last_undegraded":"2021-09-02T07:27:31.641873+0000","last_fullsized":"2021-09-02T07:27:31.641873+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_clean_scrub_stamp":"2021-09-02T07:27:30.626938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8628,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":104848972,"statfs":{"total":107374182400,"available":107365347328,"internally_reserved":0,"allocated":700416,"data_stored":618958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8556,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104849044,"statfs":{"total":107374182400,"available":107365421056,"internally_reserved":0,"allocated":692224,"data_stored":615154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7528,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":7424,"kb_avail":104850072,"statfs":{"total":107374182400,"available":107366473728,"internally_reserved":0,"allocated":98304,"data_stored":24786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7602176},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-02T07:27:36.452 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-09-02T07:27:36.733 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:36.735 INFO:teuthology.orchestra.run.smithi175.stderr:dumped all 2021-09-02T07:27:36.746 INFO:teuthology.orchestra.run.smithi175.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2021-09-02T07:27:36.584994+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24784,"kb_used_data":1464,"kb_used_omap":0,"kb_used_meta":23168,"kb_avail":314548016,"statfs":{"total":322122547200,"available":322097168384,"internally_reserved":0,"allocated":1499136,"data_stored":1262702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23724032},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000480"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-02T07:27:31.637847+0000","last_change":"2021-09-02T07:27:31.637847+0000","last_active":"2021-09-02T07:27:31.637847+0000","last_peered":"2021-09-02T07:27:31.637847+0000","last_clean":"2021-09-02T07:27:31.637847+0000","last_became_active":"2021-09-02T07:27:31.636151+0000","last_became_peered":"2021-09-02T07:27:31.636151+0000","last_unstale":"2021-09-02T07:27:31.637847+0000","last_undegraded":"2021-09-02T07:27:31.637847+0000","last_fullsized":"2021-09-02T07:27:31.637847+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_clean_scrub_stamp":"2021-09-02T07:27:30.626938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-02T07:27:31.638980+0000","last_change":"2021-09-02T07:27:31.638980+0000","last_active":"2021-09-02T07:27:31.638980+0000","last_peered":"2021-09-02T07:27:31.638980+0000","last_clean":"2021-09-02T07:27:31.638980+0000","last_became_active":"2021-09-02T07:27:31.638687+0000","last_became_peered":"2021-09-02T07:27:31.638687+0000","last_unstale":"2021-09-02T07:27:31.638980+0000","last_undegraded":"2021-09-02T07:27:31.638980+0000","last_fullsized":"2021-09-02T07:27:31.638980+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_clean_scrub_stamp":"2021-09-02T07:27:30.626938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-02T07:27:31.640043+0000","last_change":"2021-09-02T07:27:31.638452+0000","last_active":"2021-09-02T07:27:31.640043+0000","last_peered":"2021-09-02T07:27:31.640043+0000","last_clean":"2021-09-02T07:27:31.640043+0000","last_became_active":"2021-09-02T07:27:31.638158+0000","last_became_peered":"2021-09-02T07:27:31.638158+0000","last_unstale":"2021-09-02T07:27:31.640043+0000","last_undegraded":"2021-09-02T07:27:31.640043+0000","last_fullsized":"2021-09-02T07:27:31.640043+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_clean_scrub_stamp":"2021-09-02T07:27:30.626938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-02T07:27:31.636992+0000","last_change":"2021-09-02T07:27:31.636992+0000","last_active":"2021-09-02T07:27:31.636992+0000","last_peered":"2021-09-02T07:27:31.636992+0000","last_clean":"2021-09-02T07:27:31.636992+0000","last_became_active":"2021-09-02T07:27:31.636712+0000","last_became_peered":"2021-09-02T07:27:31.636712+0000","last_unstale":"2021-09-02T07:27:31.636992+0000","last_undegraded":"2021-09-02T07:27:31.636992+0000","last_fullsized":"2021-09-02T07:27:31.636992+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_clean_scrub_stamp":"2021-09-02T07:27:30.626938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-02T07:27:31.640683+0000","last_change":"2021-09-02T07:27:31.639164+0000","last_active":"2021-09-02T07:27:31.640683+0000","last_peered":"2021-09-02T07:27:31.640683+0000","last_clean":"2021-09-02T07:27:31.640683+0000","last_became_active":"2021-09-02T07:27:31.638862+0000","last_became_peered":"2021-09-02T07:27:31.638862+0000","last_unstale":"2021-09-02T07:27:31.640683+0000","last_undegraded":"2021-09-02T07:27:31.640683+0000","last_fullsized":"2021-09-02T07:27:31.640683+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_clean_scrub_stamp":"2021-09-02T07:27:30.626938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-02T07:27:32.093840+0000","last_change":"2021-09-02T07:27:31.641020+0000","last_active":"2021-09-02T07:27:32.093840+0000","last_peered":"2021-09-02T07:27:32.093840+0000","last_clean":"2021-09-02T07:27:32.093840+0000","last_became_active":"2021-09-02T07:27:31.640723+0000","last_became_peered":"2021-09-02T07:27:31.640723+0000","last_unstale":"2021-09-02T07:27:32.093840+0000","last_undegraded":"2021-09-02T07:27:32.093840+0000","last_fullsized":"2021-09-02T07:27:32.093840+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_clean_scrub_stamp":"2021-09-02T07:27:30.626938+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-02T07:27:31.637997+0000","last_change":"2021-09-02T07:27:31.637997+0000","last_active":"2021-09-02T07:27:31.637997+0000","last_peered":"2021-09-02T07:27:31.637997+0000","last_clean":"2021-09-02T07:27:31.637997+0000","last_became_active":"2021-09-02T07:27:31.637695+0000","last_became_peered":"2021-09-02T07:27:31.637695+0000","last_unstale":"2021-09-02T07:27:31.637997+0000","last_undegraded":"2021-09-02T07:27:31.637997+0000","last_fullsized":"2021-09-02T07:27:31.637997+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_clean_scrub_stamp":"2021-09-02T07:27:30.626938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-02T07:27:31.636089+0000","last_change":"2021-09-02T07:27:31.636089+0000","last_active":"2021-09-02T07:27:31.636089+0000","last_peered":"2021-09-02T07:27:31.636089+0000","last_clean":"2021-09-02T07:27:31.636089+0000","last_became_active":"2021-09-02T07:27:31.635746+0000","last_became_peered":"2021-09-02T07:27:31.635746+0000","last_unstale":"2021-09-02T07:27:31.636089+0000","last_undegraded":"2021-09-02T07:27:31.636089+0000","last_fullsized":"2021-09-02T07:27:31.636089+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_clean_scrub_stamp":"2021-09-02T07:27:30.626938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-02T07:27:31.641873+0000","last_change":"2021-09-02T07:27:31.640453+0000","last_active":"2021-09-02T07:27:31.641873+0000","last_peered":"2021-09-02T07:27:31.641873+0000","last_clean":"2021-09-02T07:27:31.641873+0000","last_became_active":"2021-09-02T07:27:31.639962+0000","last_became_peered":"2021-09-02T07:27:31.639962+0000","last_unstale":"2021-09-02T07:27:31.641873+0000","last_undegraded":"2021-09-02T07:27:31.641873+0000","last_fullsized":"2021-09-02T07:27:31.641873+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:30.626938+0000","last_clean_scrub_stamp":"2021-09-02T07:27:30.626938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8628,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":104848972,"statfs":{"total":107374182400,"available":107365347328,"internally_reserved":0,"allocated":700416,"data_stored":618958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8556,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104849044,"statfs":{"total":107374182400,"available":107365421056,"internally_reserved":0,"allocated":692224,"data_stored":615154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7600,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850000,"statfs":{"total":107374182400,"available":107366400000,"internally_reserved":0,"allocated":106496,"data_stored":28590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-02T07:27:36.748 INFO:tasks.ceph.ceph_manager.c1:clean! 2021-09-02T07:27:36.748 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2021-09-02T07:27:36.748 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2021-09-02T07:27:36.749 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2021-09-02T07:27:37.094 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:37.095 INFO:teuthology.orchestra.run.smithi175.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-09-02T07:27:37.107 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2021-09-02T07:27:37.108 INFO:teuthology.run_tasks:Running task ceph... 2021-09-02T07:27:37.122 INFO:tasks.ceph:Creating ceph cluster c2... 2021-09-02T07:27:37.122 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-09-02T07:27:37.123 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/soumyakoduri-2021-09-02_01:20:40-rgw-wip-skoduri-dbstore-object-distro-basic-smithi/6371861', 'branch': 'wip-skoduri-dbstore-object', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{centos_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '6371861', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'soumyakoduri-2021-09-02_01:20:40-rgw-wip-skoduri-dbstore-object-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.3', 'overrides': {'admin_socket': {'branch': 'wip-skoduri-dbstore-object'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9366f375ac31713c35e559edbd160565b784db1d', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '9366f375ac31713c35e559edbd160565b784db1d'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-skoduri-dbstore-object', 'sha1': '9366f375ac31713c35e559edbd160565b784db1d'}}, 'owner': 'scheduled_soumyakoduri@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '9366f375ac31713c35e559edbd160565b784db1d', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-skoduri-dbstore-object', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_9366f375ac31713c35e559edbd160565b784db1d/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '9366f375ac31713c35e559edbd160565b784db1d', 'targets': {'smithi175.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtuWE9inORipzeE+c+qLMP7PU0ecZk/wcl3RE/RP6rStzOx64hb5BodiNDj49HIXSUwRgPDulx5ybgPNn7GbpZtR7Cf9V5PRoeIXwuW8aP6+Zl7H4zba6Xqh7fQ0w3f3yM45Mx9rPUFZtxNL/PS91bh5HU6ZAv1AUKBl3tQePOsxOBg2LJqFLov7K6bG71BDb6shOEaq6f+ZUaJE7+4HX2msifQnRmuq0OXWrpxAbTmqdAbAS9CP3Aof9f1w+6g12XTkTrB2UhRBiDLgjLj6osOXyXWetUMJ9WHCyfDOhtbavgpkjQWwDH5PWwbGAhHMmlXFD/siT2vevCswIWGaWdKimy2A1o/cMcS9v2slsiPeoiQWBbBkvO3rouamC3hUEwCZYbNduTOdnngQLfzxxJ2fiBFyXuCouLGL3eP06Bei1opwh9GIjq3iSMCJVA4OJeNw2jp0Zc22mUq3LO/dgsYFygus1flReNTmTlq4s0mAKoHYFPN3NkMLktBo62dW8=', 'smithi197.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5bB3TaCFQfuTSeKh7NnjyOp1hVDtEXsXv52fZPgcyTE20csjTrt9TJJJfjAYPwU4o3iukgEiMnem+NMjtU+9SJZ2PPGOY0MwzazJCuotZ54nUPf9OaJI8lu4NVq26s7PkEWXa8+pwCmZ9hMBt7Ww216VdaAUgectXUk93NVjZAC0Nd387dL1+iT3rwaRjelbeKwkHCxyOIs+k7J3oPkZ78HKR1IlSUwHok1tdNyOWa0L7cqwoGrDslBdZ3+QYXIyMZtX8LWlRimaAjyYc53aOBn81EecHmVoSG6CxWiA94ZW+x+2/ftcObopTJWSxNP1ywgY+JtDGLjxlLKJkKNtISZ8Zg4NZEm31xV3vmXeqihw+AdX+pmHKwZeC31bmkcN/2c0bVcScBrno8DsBL3C2CgC1d3xtvaw9VcUp85m/Q8m8h2NNxGVDuKcm/BtPIKqbl+ItW7629pp0YcgNka+VKwn17KWN0fFTl+ghKKXc9BE1ynWEE4VwL6xJjeU8kds='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9366f375ac31713c35e559edbd160565b784db1d', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9366f375ac31713c35e559edbd160565b784db1d', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2021-09-02_01:20:40', 'tube': 'smithi', 'user': 'soumyakoduri', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.30581'} 2021-09-02T07:27:37.125 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-09-02T07:27:37.129 DEBUG:teuthology.orchestra.run.smithi197:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-09-02T07:27:37.149 DEBUG:teuthology.orchestra.run.smithi175:> sudo install -d -m0777 -- /var/run/ceph 2021-09-02T07:27:37.185 DEBUG:teuthology.orchestra.run.smithi197:> sudo install -d -m0777 -- /var/run/ceph 2021-09-02T07:27:37.220 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:27:37.220 DEBUG:teuthology.orchestra.run.smithi197:> dd if=/scratch_devs of=/dev/stdout 2021-09-02T07:27:37.278 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-09-02T07:27:37.282 DEBUG:teuthology.orchestra.run.smithi197:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-09-02T07:27:37.334 INFO:teuthology.orchestra.run.smithi197.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-09-02T07:27:37.334 INFO:teuthology.orchestra.run.smithi197.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-02T07:27:37.335 INFO:teuthology.orchestra.run.smithi197.stdout:Device: 6h/6d Inode: 108342 Links: 1 2021-09-02T07:27:37.335 INFO:teuthology.orchestra.run.smithi197.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-02T07:27:37.336 INFO:teuthology.orchestra.run.smithi197.stdout:Context: system_u:object_r:device_t:s0 2021-09-02T07:27:37.336 INFO:teuthology.orchestra.run.smithi197.stdout:Access: 2021-09-02 07:27:05.608053254 +0000 2021-09-02T07:27:37.336 INFO:teuthology.orchestra.run.smithi197.stdout:Modify: 2021-09-02 07:24:06.748660984 +0000 2021-09-02T07:27:37.336 INFO:teuthology.orchestra.run.smithi197.stdout:Change: 2021-09-02 07:24:06.748660984 +0000 2021-09-02T07:27:37.337 INFO:teuthology.orchestra.run.smithi197.stdout: Birth: - 2021-09-02T07:27:37.344 INFO:teuthology.orchestra.run.smithi197.stderr:1+0 records in 2021-09-02T07:27:37.345 INFO:teuthology.orchestra.run.smithi197.stderr:1+0 records out 2021-09-02T07:27:37.345 INFO:teuthology.orchestra.run.smithi197.stderr:512 bytes copied, 0.000250743 s, 2.0 MB/s 2021-09-02T07:27:37.348 DEBUG:teuthology.orchestra.run.smithi197:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-09-02T07:27:37.404 INFO:teuthology.orchestra.run.smithi197.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-09-02T07:27:37.405 INFO:teuthology.orchestra.run.smithi197.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-02T07:27:37.405 INFO:teuthology.orchestra.run.smithi197.stdout:Device: 6h/6d Inode: 104438 Links: 1 2021-09-02T07:27:37.406 INFO:teuthology.orchestra.run.smithi197.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-02T07:27:37.406 INFO:teuthology.orchestra.run.smithi197.stdout:Context: system_u:object_r:device_t:s0 2021-09-02T07:27:37.406 INFO:teuthology.orchestra.run.smithi197.stdout:Access: 2021-09-02 07:27:05.608053254 +0000 2021-09-02T07:27:37.406 INFO:teuthology.orchestra.run.smithi197.stdout:Modify: 2021-09-02 07:24:07.089656132 +0000 2021-09-02T07:27:37.407 INFO:teuthology.orchestra.run.smithi197.stdout:Change: 2021-09-02 07:24:07.089656132 +0000 2021-09-02T07:27:37.407 INFO:teuthology.orchestra.run.smithi197.stdout: Birth: - 2021-09-02T07:27:37.415 INFO:teuthology.orchestra.run.smithi197.stderr:1+0 records in 2021-09-02T07:27:37.415 INFO:teuthology.orchestra.run.smithi197.stderr:1+0 records out 2021-09-02T07:27:37.416 INFO:teuthology.orchestra.run.smithi197.stderr:512 bytes copied, 0.000287979 s, 1.8 MB/s 2021-09-02T07:27:37.417 DEBUG:teuthology.orchestra.run.smithi197:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-09-02T07:27:37.473 INFO:teuthology.orchestra.run.smithi197.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-09-02T07:27:37.473 INFO:teuthology.orchestra.run.smithi197.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-02T07:27:37.474 INFO:teuthology.orchestra.run.smithi197.stdout:Device: 6h/6d Inode: 111878 Links: 1 2021-09-02T07:27:37.474 INFO:teuthology.orchestra.run.smithi197.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-02T07:27:37.474 INFO:teuthology.orchestra.run.smithi197.stdout:Context: system_u:object_r:device_t:s0 2021-09-02T07:27:37.474 INFO:teuthology.orchestra.run.smithi197.stdout:Access: 2021-09-02 07:27:05.608053254 +0000 2021-09-02T07:27:37.474 INFO:teuthology.orchestra.run.smithi197.stdout:Modify: 2021-09-02 07:24:07.417651466 +0000 2021-09-02T07:27:37.475 INFO:teuthology.orchestra.run.smithi197.stdout:Change: 2021-09-02 07:24:07.417651466 +0000 2021-09-02T07:27:37.475 INFO:teuthology.orchestra.run.smithi197.stdout: Birth: - 2021-09-02T07:27:37.483 INFO:teuthology.orchestra.run.smithi197.stderr:1+0 records in 2021-09-02T07:27:37.483 INFO:teuthology.orchestra.run.smithi197.stderr:1+0 records out 2021-09-02T07:27:37.484 INFO:teuthology.orchestra.run.smithi197.stderr:512 bytes copied, 0.000300404 s, 1.7 MB/s 2021-09-02T07:27:37.486 DEBUG:teuthology.orchestra.run.smithi197:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-09-02T07:27:37.541 INFO:teuthology.orchestra.run.smithi197.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-09-02T07:27:37.542 INFO:teuthology.orchestra.run.smithi197.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-02T07:27:37.542 INFO:teuthology.orchestra.run.smithi197.stdout:Device: 6h/6d Inode: 107296 Links: 1 2021-09-02T07:27:37.542 INFO:teuthology.orchestra.run.smithi197.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-02T07:27:37.543 INFO:teuthology.orchestra.run.smithi197.stdout:Context: system_u:object_r:device_t:s0 2021-09-02T07:27:37.543 INFO:teuthology.orchestra.run.smithi197.stdout:Access: 2021-09-02 07:27:05.608053254 +0000 2021-09-02T07:27:37.543 INFO:teuthology.orchestra.run.smithi197.stdout:Modify: 2021-09-02 07:24:07.765646515 +0000 2021-09-02T07:27:37.543 INFO:teuthology.orchestra.run.smithi197.stdout:Change: 2021-09-02 07:24:07.765646515 +0000 2021-09-02T07:27:37.543 INFO:teuthology.orchestra.run.smithi197.stdout: Birth: - 2021-09-02T07:27:37.551 INFO:teuthology.orchestra.run.smithi197.stderr:1+0 records in 2021-09-02T07:27:37.552 INFO:teuthology.orchestra.run.smithi197.stderr:1+0 records out 2021-09-02T07:27:37.552 INFO:teuthology.orchestra.run.smithi197.stderr:512 bytes copied, 0.000196213 s, 2.6 MB/s 2021-09-02T07:27:37.554 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-02T07:27:37.554 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi197.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-02T07:27:37.555 INFO:tasks.ceph:Generating config... 2021-09-02T07:27:37.556 INFO:tasks.ceph:[client] debug rgw = 20 2021-09-02T07:27:37.556 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-09-02T07:27:37.556 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-09-02T07:27:37.557 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-02T07:27:37.557 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-09-02T07:27:37.557 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-09-02T07:27:37.558 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-09-02T07:27:37.558 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-09-02T07:27:37.558 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2021-09-02T07:27:37.559 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-09-02T07:27:37.559 INFO:tasks.ceph:[client] setgroup = ceph 2021-09-02T07:27:37.559 INFO:tasks.ceph:[client] setuser = ceph 2021-09-02T07:27:37.559 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-09-02T07:27:37.560 INFO:tasks.ceph:[mgr] debug ms = 1 2021-09-02T07:27:37.560 INFO:tasks.ceph:[mon] debug mon = 20 2021-09-02T07:27:37.560 INFO:tasks.ceph:[mon] debug ms = 1 2021-09-02T07:27:37.561 INFO:tasks.ceph:[mon] debug paxos = 20 2021-09-02T07:27:37.561 INFO:tasks.ceph:[osd] debug filestore = 20 2021-09-02T07:27:37.561 INFO:tasks.ceph:[osd] debug journal = 20 2021-09-02T07:27:37.562 INFO:tasks.ceph:[osd] debug ms = 20 2021-09-02T07:27:37.562 INFO:tasks.ceph:[osd] debug osd = 25 2021-09-02T07:27:37.562 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-09-02T07:27:37.562 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-09-02T07:27:37.563 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-09-02T07:27:37.563 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-09-02T07:27:37.563 INFO:tasks.ceph:Setting up c2.mon.a... 2021-09-02T07:27:37.564 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2021-09-02T07:27:37.657 INFO:teuthology.orchestra.run.smithi197.stdout:creating /etc/ceph/c2.keyring 2021-09-02T07:27:37.659 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2021-09-02T07:27:37.700 DEBUG:teuthology.orchestra.run.smithi197:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-09-02T07:27:37.766 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.197')] 2021-09-02T07:27:37.766 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.197'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2021-09-02T07:27:37.767 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:27:37.768 DEBUG:teuthology.orchestra.run.smithi197:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-09-02T07:27:37.821 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.197 --print /home/ubuntu/cephtest/c2.monmap 2021-09-02T07:27:37.897 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:27:37.898 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:27:37.899 INFO:teuthology.orchestra.run.smithi197.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2021-09-02T07:27:37.900 INFO:teuthology.orchestra.run.smithi197.stdout:monmaptool: generated fsid 543d2dee-3303-4f4e-93e4-af35a71cd6ea 2021-09-02T07:27:37.901 INFO:teuthology.orchestra.run.smithi197.stdout:setting min_mon_release = octopus 2021-09-02T07:27:37.901 INFO:teuthology.orchestra.run.smithi197.stdout:epoch 0 2021-09-02T07:27:37.901 INFO:teuthology.orchestra.run.smithi197.stdout:fsid 543d2dee-3303-4f4e-93e4-af35a71cd6ea 2021-09-02T07:27:37.902 INFO:teuthology.orchestra.run.smithi197.stdout:last_changed 2021-09-02T07:27:37.899129+0000 2021-09-02T07:27:37.902 INFO:teuthology.orchestra.run.smithi197.stdout:created 2021-09-02T07:27:37.899129+0000 2021-09-02T07:27:37.902 INFO:teuthology.orchestra.run.smithi197.stdout:min_mon_release 15 (octopus) 2021-09-02T07:27:37.903 INFO:teuthology.orchestra.run.smithi197.stdout:election_strategy: 1 2021-09-02T07:27:37.903 INFO:teuthology.orchestra.run.smithi197.stdout:0: [v2:172.21.15.197:3300/0,v1:172.21.15.197:6789/0] mon.a 2021-09-02T07:27:37.903 INFO:teuthology.orchestra.run.smithi197.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2021-09-02T07:27:37.904 DEBUG:teuthology.orchestra.run.smithi197:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-09-02T07:27:37.954 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 543d2dee-3303-4f4e-93e4-af35a71cd6ea... 2021-09-02T07:27:37.955 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-09-02T07:27:37.958 DEBUG:teuthology.orchestra.run.smithi197:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-09-02T07:27:38.005 INFO:teuthology.orchestra.run.smithi175.stdout:[global] 2021-09-02T07:27:38.006 INFO:teuthology.orchestra.run.smithi175.stdout: chdir = "" 2021-09-02T07:27:38.006 INFO:teuthology.orchestra.run.smithi175.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-02T07:27:38.006 INFO:teuthology.orchestra.run.smithi175.stdout: auth supported = cephx 2021-09-02T07:27:38.006 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.007 INFO:teuthology.orchestra.run.smithi175.stdout: filestore xattr use omap = true 2021-09-02T07:27:38.007 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.007 INFO:teuthology.orchestra.run.smithi175.stdout: mon clock drift allowed = 1.000 2021-09-02T07:27:38.007 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.008 INFO:teuthology.orchestra.run.smithi175.stdout: osd crush chooseleaf type = 0 2021-09-02T07:27:38.008 INFO:teuthology.orchestra.run.smithi175.stdout: auth debug = true 2021-09-02T07:27:38.008 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.008 INFO:teuthology.orchestra.run.smithi175.stdout: ms die on old message = true 2021-09-02T07:27:38.008 INFO:teuthology.orchestra.run.smithi175.stdout: ms die on bug = true 2021-09-02T07:27:38.009 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.009 INFO:teuthology.orchestra.run.smithi175.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-02T07:27:38.009 INFO:teuthology.orchestra.run.smithi175.stdout: mon pg warn max object skew = 0 2021-09-02T07:27:38.009 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.010 INFO:teuthology.orchestra.run.smithi175.stdout: # disable pg_autoscaler by default for new pools 2021-09-02T07:27:38.010 INFO:teuthology.orchestra.run.smithi175.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-02T07:27:38.010 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.010 INFO:teuthology.orchestra.run.smithi175.stdout: osd pool default size = 2 2021-09-02T07:27:38.011 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.011 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd allow primary affinity = true 2021-09-02T07:27:38.011 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd allow pg remap = true 2021-09-02T07:27:38.011 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on legacy crush tunables = false 2021-09-02T07:27:38.012 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on crush straw calc version zero = false 2021-09-02T07:27:38.012 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on no sortbitwise = false 2021-09-02T07:27:38.012 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on osd down out interval zero = false 2021-09-02T07:27:38.012 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on too few osds = false 2021-09-02T07:27:38.012 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-02T07:27:38.013 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-02T07:27:38.013 INFO:teuthology.orchestra.run.smithi175.stdout: mon_allow_pool_size_one = true 2021-09-02T07:27:38.013 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.013 INFO:teuthology.orchestra.run.smithi175.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-02T07:27:38.014 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.014 INFO:teuthology.orchestra.run.smithi175.stdout: osd default data pool replay window = 5 2021-09-02T07:27:38.014 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.015 INFO:teuthology.orchestra.run.smithi175.stdout: mon allow pool delete = true 2021-09-02T07:27:38.015 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.015 INFO:teuthology.orchestra.run.smithi175.stdout: mon cluster log file level = debug 2021-09-02T07:27:38.016 INFO:teuthology.orchestra.run.smithi175.stdout: debug asserts on shutdown = true 2021-09-02T07:27:38.016 INFO:teuthology.orchestra.run.smithi175.stdout: mon health detail to clog = false 2021-09-02T07:27:38.016 INFO:teuthology.orchestra.run.smithi175.stdout: mon host = 172.21.15.197 2021-09-02T07:27:38.016 INFO:teuthology.orchestra.run.smithi175.stdout: fsid = 543d2dee-3303-4f4e-93e4-af35a71cd6ea 2021-09-02T07:27:38.017 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.017 INFO:teuthology.orchestra.run.smithi175.stdout:[osd] 2021-09-02T07:27:38.017 INFO:teuthology.orchestra.run.smithi175.stdout: osd journal size = 100 2021-09-02T07:27:38.017 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.018 INFO:teuthology.orchestra.run.smithi175.stdout: osd scrub load threshold = 5.0 2021-09-02T07:27:38.018 INFO:teuthology.orchestra.run.smithi175.stdout: osd scrub max interval = 600 2021-09-02T07:27:38.018 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.018 INFO:teuthology.orchestra.run.smithi175.stdout: osd recover clone overlap = true 2021-09-02T07:27:38.019 INFO:teuthology.orchestra.run.smithi175.stdout: osd recovery max chunk = 1048576 2021-09-02T07:27:38.019 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.019 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug shutdown = true 2021-09-02T07:27:38.019 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug op order = true 2021-09-02T07:27:38.020 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug verify stray on activate = true 2021-09-02T07:27:38.020 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.021 INFO:teuthology.orchestra.run.smithi175.stdout: osd open classes on start = true 2021-09-02T07:27:38.021 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug pg log writeout = true 2021-09-02T07:27:38.022 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.022 INFO:teuthology.orchestra.run.smithi175.stdout: osd deep scrub update digest min age = 30 2021-09-02T07:27:38.023 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.023 INFO:teuthology.orchestra.run.smithi175.stdout: osd map max advance = 10 2021-09-02T07:27:38.023 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.024 INFO:teuthology.orchestra.run.smithi175.stdout: journal zero on create = true 2021-09-02T07:27:38.024 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.024 INFO:teuthology.orchestra.run.smithi175.stdout: filestore ondisk finisher threads = 3 2021-09-02T07:27:38.025 INFO:teuthology.orchestra.run.smithi175.stdout: filestore apply finisher threads = 3 2021-09-02T07:27:38.025 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.025 INFO:teuthology.orchestra.run.smithi175.stdout: bdev debug aio = true 2021-09-02T07:27:38.025 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug misdirected ops = true 2021-09-02T07:27:38.026 INFO:teuthology.orchestra.run.smithi175.stdout: debug filestore = 20 2021-09-02T07:27:38.026 INFO:teuthology.orchestra.run.smithi175.stdout: debug journal = 20 2021-09-02T07:27:38.027 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 20 2021-09-02T07:27:38.027 INFO:teuthology.orchestra.run.smithi175.stdout: debug osd = 25 2021-09-02T07:27:38.027 INFO:teuthology.orchestra.run.smithi175.stdout: rocksdb delete range threshold = 0 2021-09-02T07:27:38.027 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.028 INFO:teuthology.orchestra.run.smithi175.stdout:[mgr] 2021-09-02T07:27:38.028 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2021-09-02T07:27:38.028 INFO:teuthology.orchestra.run.smithi175.stdout: debug mgr = 20 2021-09-02T07:27:38.029 INFO:teuthology.orchestra.run.smithi175.stdout: debug mon = 20 2021-09-02T07:27:38.029 INFO:teuthology.orchestra.run.smithi175.stdout: debug auth = 20 2021-09-02T07:27:38.029 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min pgs per osd = 4 2021-09-02T07:27:38.029 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min bytes per osd = 10 2021-09-02T07:27:38.030 INFO:teuthology.orchestra.run.smithi175.stdout: mgr/telemetry/nag = false 2021-09-02T07:27:38.030 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.030 INFO:teuthology.orchestra.run.smithi175.stdout:[mon] 2021-09-02T07:27:38.030 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2021-09-02T07:27:38.031 INFO:teuthology.orchestra.run.smithi175.stdout: debug mon = 20 2021-09-02T07:27:38.031 INFO:teuthology.orchestra.run.smithi175.stdout: debug paxos = 20 2021-09-02T07:27:38.031 INFO:teuthology.orchestra.run.smithi175.stdout: debug auth = 20 2021-09-02T07:27:38.031 INFO:teuthology.orchestra.run.smithi175.stdout: mon data avail warn = 5 2021-09-02T07:27:38.032 INFO:teuthology.orchestra.run.smithi175.stdout: mon mgr mkfs grace = 240 2021-09-02T07:27:38.032 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min pgs per osd = 4 2021-09-02T07:27:38.032 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd reporter subtree level = osd 2021-09-02T07:27:38.032 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd prime pg temp = true 2021-09-02T07:27:38.033 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min bytes per osd = 10 2021-09-02T07:27:38.033 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.033 INFO:teuthology.orchestra.run.smithi175.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-02T07:27:38.034 INFO:teuthology.orchestra.run.smithi175.stdout: auth mon ticket ttl = 660 # 11m 2021-09-02T07:27:38.034 INFO:teuthology.orchestra.run.smithi175.stdout: auth service ticket ttl = 240 # 4m 2021-09-02T07:27:38.034 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.034 INFO:teuthology.orchestra.run.smithi175.stdout: # don't complain about insecure global_id in the test suite 2021-09-02T07:27:38.034 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-02T07:27:38.035 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-02T07:27:38.035 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:27:38.035 INFO:teuthology.orchestra.run.smithi175.stdout:[client] 2021-09-02T07:27:38.036 INFO:teuthology.orchestra.run.smithi175.stdout: rgw cache enabled = true 2021-09-02T07:27:38.036 INFO:teuthology.orchestra.run.smithi175.stdout: rgw enable ops log = true 2021-09-02T07:27:38.036 INFO:teuthology.orchestra.run.smithi175.stdout: rgw enable usage log = true 2021-09-02T07:27:38.036 INFO:teuthology.orchestra.run.smithi175.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-02T07:27:38.037 INFO:teuthology.orchestra.run.smithi175.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-02T07:27:38.037 INFO:teuthology.orchestra.run.smithi175.stdout: debug rgw = 20 2021-09-02T07:27:38.037 INFO:teuthology.orchestra.run.smithi175.stdout: rgw crypt require ssl = False 2021-09-02T07:27:38.037 INFO:teuthology.orchestra.run.smithi175.stdout: rgw crypt s3 kms backend = testing 2021-09-02T07:27:38.038 INFO:teuthology.orchestra.run.smithi175.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-02T07:27:38.038 INFO:teuthology.orchestra.run.smithi175.stdout: rgw curl low speed time = 300 2021-09-02T07:27:38.038 INFO:teuthology.orchestra.run.smithi175.stdout: rgw data log num shards = 4 2021-09-02T07:27:38.039 INFO:teuthology.orchestra.run.smithi175.stdout: rgw md log max shards = 4 2021-09-02T07:27:38.039 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync log trim interval = 0 2021-09-02T07:27:38.039 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync meta inject err probability = 0.1 2021-09-02T07:27:38.040 INFO:teuthology.orchestra.run.smithi175.stdout: rgw sync obj etag verify = True 2021-09-02T07:27:38.040 INFO:teuthology.orchestra.run.smithi175.stdout: setgroup = ceph 2021-09-02T07:27:38.040 INFO:teuthology.orchestra.run.smithi175.stdout: setuser = ceph 2021-09-02T07:27:38.041 INFO:teuthology.orchestra.run.smithi175.stdout:[mon.a] 2021-09-02T07:27:38.041 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.0] 2021-09-02T07:27:38.041 INFO:teuthology.orchestra.run.smithi175.stdout: osd_max_omap_entries_per_request = 10 2021-09-02T07:27:38.041 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.1] 2021-09-02T07:27:38.042 INFO:teuthology.orchestra.run.smithi175.stdout: osd_max_omap_entries_per_request = 1000 2021-09-02T07:27:38.042 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.2] 2021-09-02T07:27:38.042 INFO:teuthology.orchestra.run.smithi175.stdout: osd_max_omap_entries_per_request = 10000 2021-09-02T07:27:38.043 INFO:teuthology.orchestra.run.smithi197.stdout:[global] 2021-09-02T07:27:38.044 INFO:teuthology.orchestra.run.smithi197.stdout: chdir = "" 2021-09-02T07:27:38.044 INFO:teuthology.orchestra.run.smithi197.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-02T07:27:38.044 INFO:teuthology.orchestra.run.smithi197.stdout: auth supported = cephx 2021-09-02T07:27:38.044 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.045 INFO:teuthology.orchestra.run.smithi197.stdout: filestore xattr use omap = true 2021-09-02T07:27:38.045 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.045 INFO:teuthology.orchestra.run.smithi197.stdout: mon clock drift allowed = 1.000 2021-09-02T07:27:38.045 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.046 INFO:teuthology.orchestra.run.smithi197.stdout: osd crush chooseleaf type = 0 2021-09-02T07:27:38.046 INFO:teuthology.orchestra.run.smithi197.stdout: auth debug = true 2021-09-02T07:27:38.046 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.046 INFO:teuthology.orchestra.run.smithi197.stdout: ms die on old message = true 2021-09-02T07:27:38.047 INFO:teuthology.orchestra.run.smithi197.stdout: ms die on bug = true 2021-09-02T07:27:38.047 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.047 INFO:teuthology.orchestra.run.smithi197.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-02T07:27:38.047 INFO:teuthology.orchestra.run.smithi197.stdout: mon pg warn max object skew = 0 2021-09-02T07:27:38.048 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.048 INFO:teuthology.orchestra.run.smithi197.stdout: # disable pg_autoscaler by default for new pools 2021-09-02T07:27:38.048 INFO:teuthology.orchestra.run.smithi197.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-02T07:27:38.048 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.048 INFO:teuthology.orchestra.run.smithi197.stdout: osd pool default size = 2 2021-09-02T07:27:38.049 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.049 INFO:teuthology.orchestra.run.smithi197.stdout: mon osd allow primary affinity = true 2021-09-02T07:27:38.049 INFO:teuthology.orchestra.run.smithi197.stdout: mon osd allow pg remap = true 2021-09-02T07:27:38.049 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on legacy crush tunables = false 2021-09-02T07:27:38.050 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on crush straw calc version zero = false 2021-09-02T07:27:38.050 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on no sortbitwise = false 2021-09-02T07:27:38.050 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on osd down out interval zero = false 2021-09-02T07:27:38.050 INFO:teuthology.orchestra.run.smithi197.stdout: mon warn on too few osds = false 2021-09-02T07:27:38.050 INFO:teuthology.orchestra.run.smithi197.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-02T07:27:38.051 INFO:teuthology.orchestra.run.smithi197.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-02T07:27:38.051 INFO:teuthology.orchestra.run.smithi197.stdout: mon_allow_pool_size_one = true 2021-09-02T07:27:38.051 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.051 INFO:teuthology.orchestra.run.smithi197.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-02T07:27:38.052 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.052 INFO:teuthology.orchestra.run.smithi197.stdout: osd default data pool replay window = 5 2021-09-02T07:27:38.052 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.052 INFO:teuthology.orchestra.run.smithi197.stdout: mon allow pool delete = true 2021-09-02T07:27:38.053 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.053 INFO:teuthology.orchestra.run.smithi197.stdout: mon cluster log file level = debug 2021-09-02T07:27:38.053 INFO:teuthology.orchestra.run.smithi197.stdout: debug asserts on shutdown = true 2021-09-02T07:27:38.053 INFO:teuthology.orchestra.run.smithi197.stdout: mon health detail to clog = false 2021-09-02T07:27:38.053 INFO:teuthology.orchestra.run.smithi197.stdout: mon host = 172.21.15.197 2021-09-02T07:27:38.054 INFO:teuthology.orchestra.run.smithi197.stdout: fsid = 543d2dee-3303-4f4e-93e4-af35a71cd6ea 2021-09-02T07:27:38.054 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.054 INFO:teuthology.orchestra.run.smithi197.stdout:[osd] 2021-09-02T07:27:38.054 INFO:teuthology.orchestra.run.smithi197.stdout: osd journal size = 100 2021-09-02T07:27:38.055 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.055 INFO:teuthology.orchestra.run.smithi197.stdout: osd scrub load threshold = 5.0 2021-09-02T07:27:38.055 INFO:teuthology.orchestra.run.smithi197.stdout: osd scrub max interval = 600 2021-09-02T07:27:38.055 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.056 INFO:teuthology.orchestra.run.smithi197.stdout: osd recover clone overlap = true 2021-09-02T07:27:38.056 INFO:teuthology.orchestra.run.smithi197.stdout: osd recovery max chunk = 1048576 2021-09-02T07:27:38.056 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.056 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug shutdown = true 2021-09-02T07:27:38.056 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug op order = true 2021-09-02T07:27:38.057 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug verify stray on activate = true 2021-09-02T07:27:38.057 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.057 INFO:teuthology.orchestra.run.smithi197.stdout: osd open classes on start = true 2021-09-02T07:27:38.057 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug pg log writeout = true 2021-09-02T07:27:38.058 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.058 INFO:teuthology.orchestra.run.smithi197.stdout: osd deep scrub update digest min age = 30 2021-09-02T07:27:38.058 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.058 INFO:teuthology.orchestra.run.smithi197.stdout: osd map max advance = 10 2021-09-02T07:27:38.058 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.059 INFO:teuthology.orchestra.run.smithi197.stdout: journal zero on create = true 2021-09-02T07:27:38.059 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.059 INFO:teuthology.orchestra.run.smithi197.stdout: filestore ondisk finisher threads = 3 2021-09-02T07:27:38.059 INFO:teuthology.orchestra.run.smithi197.stdout: filestore apply finisher threads = 3 2021-09-02T07:27:38.060 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.060 INFO:teuthology.orchestra.run.smithi197.stdout: bdev debug aio = true 2021-09-02T07:27:38.060 INFO:teuthology.orchestra.run.smithi197.stdout: osd debug misdirected ops = true 2021-09-02T07:27:38.060 INFO:teuthology.orchestra.run.smithi197.stdout: debug filestore = 20 2021-09-02T07:27:38.061 INFO:teuthology.orchestra.run.smithi197.stdout: debug journal = 20 2021-09-02T07:27:38.061 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 20 2021-09-02T07:27:38.061 INFO:teuthology.orchestra.run.smithi197.stdout: debug osd = 25 2021-09-02T07:27:38.061 INFO:teuthology.orchestra.run.smithi197.stdout: rocksdb delete range threshold = 0 2021-09-02T07:27:38.061 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.062 INFO:teuthology.orchestra.run.smithi197.stdout:[mgr] 2021-09-02T07:27:38.062 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 1 2021-09-02T07:27:38.062 INFO:teuthology.orchestra.run.smithi197.stdout: debug mgr = 20 2021-09-02T07:27:38.062 INFO:teuthology.orchestra.run.smithi197.stdout: debug mon = 20 2021-09-02T07:27:38.063 INFO:teuthology.orchestra.run.smithi197.stdout: debug auth = 20 2021-09-02T07:27:38.063 INFO:teuthology.orchestra.run.smithi197.stdout: mon reweight min pgs per osd = 4 2021-09-02T07:27:38.063 INFO:teuthology.orchestra.run.smithi197.stdout: mon reweight min bytes per osd = 10 2021-09-02T07:27:38.063 INFO:teuthology.orchestra.run.smithi197.stdout: mgr/telemetry/nag = false 2021-09-02T07:27:38.064 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.064 INFO:teuthology.orchestra.run.smithi197.stdout:[mon] 2021-09-02T07:27:38.064 INFO:teuthology.orchestra.run.smithi197.stdout: debug ms = 1 2021-09-02T07:27:38.064 INFO:teuthology.orchestra.run.smithi197.stdout: debug mon = 20 2021-09-02T07:27:38.065 INFO:teuthology.orchestra.run.smithi197.stdout: debug paxos = 20 2021-09-02T07:27:38.065 INFO:teuthology.orchestra.run.smithi197.stdout: debug auth = 20 2021-09-02T07:27:38.065 INFO:teuthology.orchestra.run.smithi197.stdout: mon data avail warn = 5 2021-09-02T07:27:38.065 INFO:teuthology.orchestra.run.smithi197.stdout: mon mgr mkfs grace = 240 2021-09-02T07:27:38.065 INFO:teuthology.orchestra.run.smithi197.stdout: mon reweight min pgs per osd = 4 2021-09-02T07:27:38.066 INFO:teuthology.orchestra.run.smithi197.stdout: mon osd reporter subtree level = osd 2021-09-02T07:27:38.066 INFO:teuthology.orchestra.run.smithi197.stdout: mon osd prime pg temp = true 2021-09-02T07:27:38.066 INFO:teuthology.orchestra.run.smithi197.stdout: mon reweight min bytes per osd = 10 2021-09-02T07:27:38.066 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.067 INFO:teuthology.orchestra.run.smithi197.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-02T07:27:38.067 INFO:teuthology.orchestra.run.smithi197.stdout: auth mon ticket ttl = 660 # 11m 2021-09-02T07:27:38.067 INFO:teuthology.orchestra.run.smithi197.stdout: auth service ticket ttl = 240 # 4m 2021-09-02T07:27:38.067 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.068 INFO:teuthology.orchestra.run.smithi197.stdout: # don't complain about insecure global_id in the test suite 2021-09-02T07:27:38.068 INFO:teuthology.orchestra.run.smithi197.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-02T07:27:38.068 INFO:teuthology.orchestra.run.smithi197.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-02T07:27:38.068 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:38.068 INFO:teuthology.orchestra.run.smithi197.stdout:[client] 2021-09-02T07:27:38.069 INFO:teuthology.orchestra.run.smithi197.stdout: rgw cache enabled = true 2021-09-02T07:27:38.069 INFO:teuthology.orchestra.run.smithi197.stdout: rgw enable ops log = true 2021-09-02T07:27:38.069 INFO:teuthology.orchestra.run.smithi197.stdout: rgw enable usage log = true 2021-09-02T07:27:38.069 INFO:teuthology.orchestra.run.smithi197.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-02T07:27:38.070 INFO:teuthology.orchestra.run.smithi197.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-02T07:27:38.070 INFO:teuthology.orchestra.run.smithi197.stdout: debug rgw = 20 2021-09-02T07:27:38.070 INFO:teuthology.orchestra.run.smithi197.stdout: rgw crypt require ssl = False 2021-09-02T07:27:38.070 INFO:teuthology.orchestra.run.smithi197.stdout: rgw crypt s3 kms backend = testing 2021-09-02T07:27:38.071 INFO:teuthology.orchestra.run.smithi197.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-02T07:27:38.071 INFO:teuthology.orchestra.run.smithi197.stdout: rgw curl low speed time = 300 2021-09-02T07:27:38.071 INFO:teuthology.orchestra.run.smithi197.stdout: rgw data log num shards = 4 2021-09-02T07:27:38.071 INFO:teuthology.orchestra.run.smithi197.stdout: rgw md log max shards = 4 2021-09-02T07:27:38.072 INFO:teuthology.orchestra.run.smithi197.stdout: rgw sync log trim interval = 0 2021-09-02T07:27:38.072 INFO:teuthology.orchestra.run.smithi197.stdout: rgw sync meta inject err probability = 0.1 2021-09-02T07:27:38.072 INFO:teuthology.orchestra.run.smithi197.stdout: rgw sync obj etag verify = True 2021-09-02T07:27:38.072 INFO:teuthology.orchestra.run.smithi197.stdout: setgroup = ceph 2021-09-02T07:27:38.073 INFO:teuthology.orchestra.run.smithi197.stdout: setuser = ceph 2021-09-02T07:27:38.073 INFO:teuthology.orchestra.run.smithi197.stdout:[mon.a] 2021-09-02T07:27:38.073 INFO:teuthology.orchestra.run.smithi197.stdout:[osd.0] 2021-09-02T07:27:38.073 INFO:teuthology.orchestra.run.smithi197.stdout: osd_max_omap_entries_per_request = 10 2021-09-02T07:27:38.073 INFO:teuthology.orchestra.run.smithi197.stdout:[osd.1] 2021-09-02T07:27:38.074 INFO:teuthology.orchestra.run.smithi197.stdout: osd_max_omap_entries_per_request = 1000 2021-09-02T07:27:38.074 INFO:teuthology.orchestra.run.smithi197.stdout:[osd.2] 2021-09-02T07:27:38.074 INFO:teuthology.orchestra.run.smithi197.stdout: osd_max_omap_entries_per_request = 10000 2021-09-02T07:27:38.076 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2021-09-02T07:27:38.076 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2021-09-02T07:27:38.165 INFO:tasks.ceph:Copying monmap to all nodes... 2021-09-02T07:27:38.166 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:27:38.166 DEBUG:teuthology.orchestra.run.smithi197:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2021-09-02T07:27:38.180 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:27:38.181 DEBUG:teuthology.orchestra.run.smithi197:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2021-09-02T07:27:38.234 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi175.front.sepia.ceph.com 2021-09-02T07:27:38.235 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:27:38.235 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/ceph/c2.keyring 2021-09-02T07:27:38.235 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-09-02T07:27:38.275 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2021-09-02T07:27:38.276 DEBUG:teuthology.orchestra.run.smithi175:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-09-02T07:27:38.332 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi197.front.sepia.ceph.com 2021-09-02T07:27:38.333 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:27:38.333 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd of=/etc/ceph/c2.keyring 2021-09-02T07:27:38.333 DEBUG:teuthology.orchestra.run.smithi197:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-09-02T07:27:38.365 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:27:38.366 DEBUG:teuthology.orchestra.run.smithi197:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-09-02T07:27:38.420 INFO:tasks.ceph:Setting up mon nodes... 2021-09-02T07:27:38.421 INFO:tasks.ceph:Setting up mgr nodes... 2021-09-02T07:27:38.421 DEBUG:teuthology.orchestra.run.smithi197:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2021-09-02T07:27:38.506 INFO:teuthology.orchestra.run.smithi197.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2021-09-02T07:27:38.508 INFO:tasks.ceph:Setting up mds nodes... 2021-09-02T07:27:38.509 INFO:tasks.ceph_client:Setting up client nodes... 2021-09-02T07:27:38.509 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2021-09-02T07:27:38.545 INFO:teuthology.orchestra.run.smithi197.stdout:creating /etc/ceph/c2.client.0.keyring 2021-09-02T07:27:38.557 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2021-09-02T07:27:38.633 INFO:teuthology.orchestra.run.smithi197.stdout:creating /etc/ceph/c2.client.1.keyring 2021-09-02T07:27:38.645 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-09-02T07:27:38.646 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi175.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi197.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-02T07:27:38.646 DEBUG:teuthology.orchestra.run.smithi197:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2021-09-02T07:27:38.709 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-02T07:27:38.709 INFO:tasks.ceph:role: c2.osd.0 2021-09-02T07:27:38.709 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi197.front.sepia.ceph.com 2021-09-02T07:27:38.710 DEBUG:teuthology.orchestra.run.smithi197:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-09-02T07:27:38.780 INFO:teuthology.orchestra.run.smithi197.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-09-02T07:27:38.780 INFO:teuthology.orchestra.run.smithi197.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-02T07:27:38.781 INFO:teuthology.orchestra.run.smithi197.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-02T07:27:38.781 INFO:teuthology.orchestra.run.smithi197.stdout: = reflink=1 2021-09-02T07:27:38.781 INFO:teuthology.orchestra.run.smithi197.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-02T07:27:38.782 INFO:teuthology.orchestra.run.smithi197.stdout: = sunit=0 swidth=0 blks 2021-09-02T07:27:38.782 INFO:teuthology.orchestra.run.smithi197.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-02T07:27:38.782 INFO:teuthology.orchestra.run.smithi197.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-02T07:27:38.782 INFO:teuthology.orchestra.run.smithi197.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-02T07:27:38.782 INFO:teuthology.orchestra.run.smithi197.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-02T07:27:39.143 INFO:teuthology.orchestra.run.smithi197.stdout:Discarding blocks...Done. 2021-09-02T07:27:39.145 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi197.front.sepia.ceph.com -o noatime 2021-09-02T07:27:39.145 DEBUG:teuthology.orchestra.run.smithi197:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2021-09-02T07:27:39.272 DEBUG:teuthology.orchestra.run.smithi197:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2021-09-02T07:27:39.300 DEBUG:teuthology.orchestra.run.smithi197:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2021-09-02T07:27:39.364 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-02T07:27:39.364 INFO:tasks.ceph:role: c2.osd.1 2021-09-02T07:27:39.364 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi197.front.sepia.ceph.com 2021-09-02T07:27:39.365 DEBUG:teuthology.orchestra.run.smithi197:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-09-02T07:27:39.434 INFO:teuthology.orchestra.run.smithi197.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-09-02T07:27:39.434 INFO:teuthology.orchestra.run.smithi197.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-02T07:27:39.434 INFO:teuthology.orchestra.run.smithi197.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-02T07:27:39.435 INFO:teuthology.orchestra.run.smithi197.stdout: = reflink=1 2021-09-02T07:27:39.435 INFO:teuthology.orchestra.run.smithi197.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-02T07:27:39.435 INFO:teuthology.orchestra.run.smithi197.stdout: = sunit=0 swidth=0 blks 2021-09-02T07:27:39.435 INFO:teuthology.orchestra.run.smithi197.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-02T07:27:39.436 INFO:teuthology.orchestra.run.smithi197.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-02T07:27:39.436 INFO:teuthology.orchestra.run.smithi197.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-02T07:27:39.436 INFO:teuthology.orchestra.run.smithi197.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-02T07:27:39.796 INFO:teuthology.orchestra.run.smithi197.stdout:Discarding blocks...Done. 2021-09-02T07:27:39.797 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi197.front.sepia.ceph.com -o noatime 2021-09-02T07:27:39.798 DEBUG:teuthology.orchestra.run.smithi197:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2021-09-02T07:27:39.883 DEBUG:teuthology.orchestra.run.smithi197:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2021-09-02T07:27:39.910 DEBUG:teuthology.orchestra.run.smithi197:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2021-09-02T07:27:39.975 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-02T07:27:39.975 INFO:tasks.ceph:role: c2.osd.2 2021-09-02T07:27:39.975 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi197.front.sepia.ceph.com 2021-09-02T07:27:39.976 DEBUG:teuthology.orchestra.run.smithi197:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-09-02T07:27:40.045 INFO:teuthology.orchestra.run.smithi197.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-09-02T07:27:40.045 INFO:teuthology.orchestra.run.smithi197.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-02T07:27:40.045 INFO:teuthology.orchestra.run.smithi197.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-02T07:27:40.046 INFO:teuthology.orchestra.run.smithi197.stdout: = reflink=1 2021-09-02T07:27:40.046 INFO:teuthology.orchestra.run.smithi197.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-02T07:27:40.046 INFO:teuthology.orchestra.run.smithi197.stdout: = sunit=0 swidth=0 blks 2021-09-02T07:27:40.047 INFO:teuthology.orchestra.run.smithi197.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-02T07:27:40.047 INFO:teuthology.orchestra.run.smithi197.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-02T07:27:40.047 INFO:teuthology.orchestra.run.smithi197.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-02T07:27:40.048 INFO:teuthology.orchestra.run.smithi197.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-02T07:27:40.408 INFO:teuthology.orchestra.run.smithi197.stdout:Discarding blocks...Done. 2021-09-02T07:27:40.410 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi197.front.sepia.ceph.com -o noatime 2021-09-02T07:27:40.410 DEBUG:teuthology.orchestra.run.smithi197:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2021-09-02T07:27:40.506 DEBUG:teuthology.orchestra.run.smithi197:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2021-09-02T07:27:40.533 DEBUG:teuthology.orchestra.run.smithi197:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2021-09-02T07:27:40.623 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:27:40.624+0000 7f7def9770c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2021-09-02T07:27:40.623 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:27:40.625+0000 7f7def9770c0 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2021-09-02T07:27:40.624 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:27:40.625+0000 7f7def9770c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-09-02T07:27:40.624 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:27:40.625+0000 7f7def9770c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-09-02T07:27:40.624 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:27:40.625+0000 7f7def9770c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-09-02T07:27:40.625 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:27:40.626+0000 7f7def9770c0 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2021-09-02T07:27:42.705 DEBUG:teuthology.orchestra.run.smithi197:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2021-09-02T07:27:42.737 DEBUG:teuthology.orchestra.run.smithi197:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2021-09-02T07:27:42.821 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:27:42.823+0000 7f8f1df250c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2021-09-02T07:27:42.821 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:27:42.823+0000 7f8f1df250c0 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2021-09-02T07:27:42.822 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:27:42.823+0000 7f8f1df250c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-09-02T07:27:42.822 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:27:42.823+0000 7f8f1df250c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-09-02T07:27:42.822 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:27:42.823+0000 7f8f1df250c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-09-02T07:27:42.822 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:27:42.824+0000 7f8f1df250c0 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2021-09-02T07:27:44.912 DEBUG:teuthology.orchestra.run.smithi197:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2021-09-02T07:27:44.981 DEBUG:teuthology.orchestra.run.smithi197:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2021-09-02T07:27:45.065 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:27:45.066+0000 7fa0b1c5d0c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2021-09-02T07:27:45.065 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:27:45.066+0000 7fa0b1c5d0c0 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2021-09-02T07:27:45.065 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:27:45.066+0000 7fa0b1c5d0c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-09-02T07:27:45.066 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:27:45.066+0000 7fa0b1c5d0c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-09-02T07:27:45.066 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:27:45.066+0000 7fa0b1c5d0c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-09-02T07:27:45.066 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:27:45.067+0000 7fa0b1c5d0c0 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2021-09-02T07:27:47.150 DEBUG:teuthology.orchestra.run.smithi197:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2021-09-02T07:27:47.219 INFO:tasks.ceph:Reading keys from all nodes... 2021-09-02T07:27:47.220 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:27:47.220 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2021-09-02T07:27:47.286 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:27:47.287 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2021-09-02T07:27:47.353 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:27:47.353 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2021-09-02T07:27:47.419 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:27:47.419 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2021-09-02T07:27:47.484 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:27:47.485 DEBUG:teuthology.orchestra.run.smithi197:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2021-09-02T07:27:47.539 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:27:47.540 DEBUG:teuthology.orchestra.run.smithi197:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2021-09-02T07:27:47.594 INFO:tasks.ceph:Adding keys to all mons... 2021-09-02T07:27:47.595 DEBUG:teuthology.orchestra.run.smithi197:> sudo tee -a /etc/ceph/c2.keyring 2021-09-02T07:27:47.656 INFO:teuthology.orchestra.run.smithi197.stdout:[mgr.x] 2021-09-02T07:27:47.656 INFO:teuthology.orchestra.run.smithi197.stdout: key = AQDqfDBhT8BgHhAAB+UaS8LIG53YNkx3UxjdLw== 2021-09-02T07:27:47.657 INFO:teuthology.orchestra.run.smithi197.stdout:[osd.0] 2021-09-02T07:27:47.657 INFO:teuthology.orchestra.run.smithi197.stdout: key = AQDsfDBhs0I7JRAAEUWVJmQWknfTmzkj1L8roQ== 2021-09-02T07:27:47.657 INFO:teuthology.orchestra.run.smithi197.stdout:[osd.1] 2021-09-02T07:27:47.657 INFO:teuthology.orchestra.run.smithi197.stdout: key = AQDufDBhdAEdMRAAGyYSvmV0H7+e0igSo1NhmQ== 2021-09-02T07:27:47.657 INFO:teuthology.orchestra.run.smithi197.stdout:[osd.2] 2021-09-02T07:27:47.658 INFO:teuthology.orchestra.run.smithi197.stdout: key = AQDxfDBhMGEBBBAA71D7FUGVCgzKvYZ/BIXdAg== 2021-09-02T07:27:47.658 INFO:teuthology.orchestra.run.smithi197.stdout:[client.0] 2021-09-02T07:27:47.658 INFO:teuthology.orchestra.run.smithi197.stdout: key = AQDqfDBh8Wi4IBAAy46ywSlqZ0ldB9WE+RhyJQ== 2021-09-02T07:27:47.658 INFO:teuthology.orchestra.run.smithi197.stdout:[client.1] 2021-09-02T07:27:47.659 INFO:teuthology.orchestra.run.smithi197.stdout: key = AQDqfDBhvfvyJRAAwALTLuioDB/2xfwAruPNEg== 2021-09-02T07:27:47.660 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-09-02T07:27:47.745 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-09-02T07:27:47.791 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-09-02T07:27:47.837 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-09-02T07:27:47.882 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-02T07:27:47.971 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-02T07:27:48.020 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-09-02T07:27:48.021 DEBUG:teuthology.orchestra.run.smithi197:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2021-09-02T07:27:48.044 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2021-09-02T07:27:48.134 DEBUG:teuthology.orchestra.run.smithi197:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2021-09-02T07:27:48.160 DEBUG:teuthology.orchestra.run.smithi197:> rm -- /home/ubuntu/cephtest/c2.monmap 2021-09-02T07:27:48.214 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2021-09-02T07:27:48.215 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2021-09-02T07:27:48.216 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2021-09-02T07:27:48.257 INFO:tasks.ceph.c2.mon.a:Started 2021-09-02T07:27:48.257 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2021-09-02T07:27:48.258 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2021-09-02T07:27:48.258 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2021-09-02T07:27:48.260 INFO:tasks.ceph.c2.mgr.x:Started 2021-09-02T07:27:48.261 INFO:tasks.ceph:Setting crush tunables to default 2021-09-02T07:27:48.261 DEBUG:teuthology.orchestra.run.smithi197:> sudo ceph --cluster c2 osd crush tunables default 2021-09-02T07:27:48.455 INFO:teuthology.orchestra.run.smithi197.stderr:adjusted tunables profile to default 2021-09-02T07:27:48.465 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2021-09-02T07:27:48.466 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:27:48.466 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2021-09-02T07:27:48.490 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:27:48.490 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2021-09-02T07:27:48.554 DEBUG:teuthology.orchestra.run.smithi197:> set -ex 2021-09-02T07:27:48.554 DEBUG:teuthology.orchestra.run.smithi197:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2021-09-02T07:27:48.619 DEBUG:teuthology.orchestra.run.smithi197:> sudo ceph --cluster c2 osd new e0e9de1e-b652-4e2e-9dbf-8ab997b40365 0 2021-09-02T07:27:48.858 INFO:teuthology.orchestra.run.smithi197.stdout:0 2021-09-02T07:27:48.870 DEBUG:teuthology.orchestra.run.smithi197:> sudo ceph --cluster c2 osd new 56342390-0149-463b-bca1-27bdeb4d6d73 1 2021-09-02T07:27:49.073 INFO:teuthology.orchestra.run.smithi197.stdout:1 2021-09-02T07:27:49.086 DEBUG:teuthology.orchestra.run.smithi197:> sudo ceph --cluster c2 osd new b1cdb47e-f589-4df4-b22d-585e435c01f7 2 2021-09-02T07:27:49.284 INFO:teuthology.orchestra.run.smithi197.stdout:2 2021-09-02T07:27:49.296 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2021-09-02T07:27:49.297 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2021-09-02T07:27:49.299 INFO:tasks.ceph.c2.osd.0:Started 2021-09-02T07:27:49.300 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2021-09-02T07:27:49.301 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2021-09-02T07:27:49.304 INFO:tasks.ceph.c2.osd.1:Started 2021-09-02T07:27:49.304 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2021-09-02T07:27:49.304 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2021-09-02T07:27:49.306 INFO:tasks.ceph.c2.osd.2:Started 2021-09-02T07:27:49.307 INFO:tasks.ceph:Waiting for OSDs to come up 2021-09-02T07:27:49.610 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-09-02T07:27:49.793 INFO:teuthology.misc.health.smithi197.stdout: 2021-09-02T07:27:49.793 INFO:teuthology.misc.health.smithi197.stdout:{"epoch":5,"fsid":"543d2dee-3303-4f4e-93e4-af35a71cd6ea","created":"2021-09-02T07:27:48.357746+0000","modified":"2021-09-02T07:27:49.285665+0000","last_up_change":"0.000000","last_in_change":"2021-09-02T07:27:49.285665+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"e0e9de1e-b652-4e2e-9dbf-8ab997b40365","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"56342390-0149-463b-bca1-27bdeb4d6d73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b1cdb47e-f589-4df4-b22d-585e435c01f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-02T07:27:49.801 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-09-02T07:27:49.889 INFO:tasks.ceph.c2.osd.1.smithi197.stderr:2021-09-02T07:27:49.891+0000 7f18945c80c0 -1 Falling back to public interface 2021-09-02T07:27:49.897 INFO:tasks.ceph.c2.osd.0.smithi197.stderr:2021-09-02T07:27:49.899+0000 7fc6300c30c0 -1 Falling back to public interface 2021-09-02T07:27:49.902 INFO:tasks.ceph.c2.osd.2.smithi197.stderr:2021-09-02T07:27:49.904+0000 7f0f2df3c0c0 -1 Falling back to public interface 2021-09-02T07:27:52.705 INFO:tasks.ceph.c2.osd.0.smithi197.stderr:2021-09-02T07:27:52.706+0000 7fc6300c30c0 -1 osd.0 0 log_to_monitors {default=true} 2021-09-02T07:27:52.705 INFO:tasks.ceph.c2.osd.1.smithi197.stderr:2021-09-02T07:27:52.707+0000 7f18945c80c0 -1 osd.1 0 log_to_monitors {default=true} 2021-09-02T07:27:52.706 INFO:tasks.ceph.c2.osd.2.smithi197.stderr:2021-09-02T07:27:52.707+0000 7f0f2df3c0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-09-02T07:27:55.159 INFO:tasks.ceph.c2.osd.1.smithi197.stderr:2021-09-02T07:27:55.160+0000 7f188af86700 -1 osd.1 0 waiting for initial osdmap 2021-09-02T07:27:55.161 INFO:tasks.ceph.c2.osd.1.smithi197.stderr:2021-09-02T07:27:55.162+0000 7f1883919700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-02T07:27:55.227 INFO:tasks.ceph.c2.osd.0.smithi197.stderr:2021-09-02T07:27:55.229+0000 7fc626a81700 -1 osd.0 0 waiting for initial osdmap 2021-09-02T07:27:55.229 INFO:tasks.ceph.c2.osd.0.smithi197.stderr:2021-09-02T07:27:55.230+0000 7fc61f414700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-02T07:27:55.483 INFO:tasks.ceph.c2.osd.2.smithi197.stderr:2021-09-02T07:27:55.484+0000 7f0f248fa700 -1 osd.2 0 waiting for initial osdmap 2021-09-02T07:27:55.485 INFO:tasks.ceph.c2.osd.2.smithi197.stderr:2021-09-02T07:27:55.486+0000 7f0f1d28d700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-02T07:27:56.105 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-09-02T07:27:56.389 INFO:teuthology.misc.health.smithi197.stdout: 2021-09-02T07:27:56.389 INFO:teuthology.misc.health.smithi197.stdout:{"epoch":8,"fsid":"543d2dee-3303-4f4e-93e4-af35a71cd6ea","created":"2021-09-02T07:27:48.357746+0000","modified":"2021-09-02T07:27:56.159818+0000","last_up_change":"2021-09-02T07:27:56.159818+0000","last_in_change":"2021-09-02T07:27:49.285665+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"e0e9de1e-b652-4e2e-9dbf-8ab997b40365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6808","nonce":30570},{"type":"v1","addr":"172.21.15.197:6809","nonce":30570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6810","nonce":30570},{"type":"v1","addr":"172.21.15.197:6811","nonce":30570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6814","nonce":30570},{"type":"v1","addr":"172.21.15.197:6815","nonce":30570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6812","nonce":30570},{"type":"v1","addr":"172.21.15.197:6813","nonce":30570}]},"public_addr":"172.21.15.197:6809/30570","cluster_addr":"172.21.15.197:6811/30570","heartbeat_back_addr":"172.21.15.197:6815/30570","heartbeat_front_addr":"172.21.15.197:6813/30570","state":["exists","up"]},{"osd":1,"uuid":"56342390-0149-463b-bca1-27bdeb4d6d73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6800","nonce":30571},{"type":"v1","addr":"172.21.15.197:6801","nonce":30571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6802","nonce":30571},{"type":"v1","addr":"172.21.15.197:6803","nonce":30571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6806","nonce":30571},{"type":"v1","addr":"172.21.15.197:6807","nonce":30571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6804","nonce":30571},{"type":"v1","addr":"172.21.15.197:6805","nonce":30571}]},"public_addr":"172.21.15.197:6801/30571","cluster_addr":"172.21.15.197:6803/30571","heartbeat_back_addr":"172.21.15.197:6807/30571","heartbeat_front_addr":"172.21.15.197:6805/30571","state":["exists","up"]},{"osd":2,"uuid":"b1cdb47e-f589-4df4-b22d-585e435c01f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6816","nonce":30572},{"type":"v1","addr":"172.21.15.197:6817","nonce":30572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6818","nonce":30572},{"type":"v1","addr":"172.21.15.197:6819","nonce":30572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6822","nonce":30572},{"type":"v1","addr":"172.21.15.197:6823","nonce":30572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6820","nonce":30572},{"type":"v1","addr":"172.21.15.197:6821","nonce":30572}]},"public_addr":"172.21.15.197:6817/30572","cluster_addr":"172.21.15.197:6819/30572","heartbeat_back_addr":"172.21.15.197:6823/30572","heartbeat_front_addr":"172.21.15.197:6821/30572","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-02T07:27:56.401 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-09-02T07:27:56.401 INFO:tasks.ceph:Creating RBD pool 2021-09-02T07:27:56.402 DEBUG:teuthology.orchestra.run.smithi197:> sudo ceph --cluster c2 osd pool create rbd 8 2021-09-02T07:27:57.317 INFO:teuthology.orchestra.run.smithi197.stderr:pool 'rbd' created 2021-09-02T07:27:57.329 DEBUG:teuthology.orchestra.run.smithi197:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-09-02T07:27:58.322 INFO:teuthology.orchestra.run.smithi197.stderr:enabled application 'rbd' on pool 'rbd' 2021-09-02T07:27:58.336 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2021-09-02T07:27:58.337 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-09-02T07:27:58.337 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-09-02T07:27:58.655 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:58.656 INFO:teuthology.orchestra.run.smithi197.stdout:{"epoch":10,"fsid":"543d2dee-3303-4f4e-93e4-af35a71cd6ea","created":"2021-09-02T07:27:48.357746+0000","modified":"2021-09-02T07:27:58.313484+0000","last_up_change":"2021-09-02T07:27:56.159818+0000","last_in_change":"2021-09-02T07:27:49.285665+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-02T07:27:56.700049+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-02T07:27:57.303114+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"e0e9de1e-b652-4e2e-9dbf-8ab997b40365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6808","nonce":30570},{"type":"v1","addr":"172.21.15.197:6809","nonce":30570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6810","nonce":30570},{"type":"v1","addr":"172.21.15.197:6811","nonce":30570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6814","nonce":30570},{"type":"v1","addr":"172.21.15.197:6815","nonce":30570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6812","nonce":30570},{"type":"v1","addr":"172.21.15.197:6813","nonce":30570}]},"public_addr":"172.21.15.197:6809/30570","cluster_addr":"172.21.15.197:6811/30570","heartbeat_back_addr":"172.21.15.197:6815/30570","heartbeat_front_addr":"172.21.15.197:6813/30570","state":["exists","up"]},{"osd":1,"uuid":"56342390-0149-463b-bca1-27bdeb4d6d73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6800","nonce":30571},{"type":"v1","addr":"172.21.15.197:6801","nonce":30571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6802","nonce":30571},{"type":"v1","addr":"172.21.15.197:6803","nonce":30571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6806","nonce":30571},{"type":"v1","addr":"172.21.15.197:6807","nonce":30571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6804","nonce":30571},{"type":"v1","addr":"172.21.15.197:6805","nonce":30571}]},"public_addr":"172.21.15.197:6801/30571","cluster_addr":"172.21.15.197:6803/30571","heartbeat_back_addr":"172.21.15.197:6807/30571","heartbeat_front_addr":"172.21.15.197:6805/30571","state":["exists","up"]},{"osd":2,"uuid":"b1cdb47e-f589-4df4-b22d-585e435c01f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6816","nonce":30572},{"type":"v1","addr":"172.21.15.197:6817","nonce":30572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6818","nonce":30572},{"type":"v1","addr":"172.21.15.197:6819","nonce":30572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6822","nonce":30572},{"type":"v1","addr":"172.21.15.197:6823","nonce":30572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6820","nonce":30572},{"type":"v1","addr":"172.21.15.197:6821","nonce":30572}]},"public_addr":"172.21.15.197:6817/30572","cluster_addr":"172.21.15.197:6819/30572","heartbeat_back_addr":"172.21.15.197:6823/30572","heartbeat_front_addr":"172.21.15.197:6821/30572","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-02T07:27:53.664089+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-02T07:27:53.706923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-02T07:27:53.738200+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-02T07:27:58.665 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-09-02T07:27:56.700049+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2021-09-02T07:27:57.303114+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2021-09-02T07:27:58.666 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get rbd pg_num 2021-09-02T07:27:58.958 INFO:teuthology.orchestra.run.smithi197.stdout:pg_num: 8 2021-09-02T07:27:58.971 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get .mgr pg_num 2021-09-02T07:27:59.254 INFO:teuthology.orchestra.run.smithi197.stdout:pg_num: 1 2021-09-02T07:27:59.267 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2021-09-02T07:27:59.267 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2021-09-02T07:27:59.267 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2021-09-02T07:27:59.597 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:59.609 INFO:teuthology.orchestra.run.smithi197.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6824","nonce":30253},{"type":"v1","addr":"172.21.15.197:6825","nonce":30253}]},"active_addr":"172.21.15.197:6825/30253","active_change":"2021-09-02T07:27:53.269363+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"autoscale_profile":{"name":"autoscale_profile","type":"str","level":"advanced","flags":1,"default_value":"scale-up","min":"","max":"","enum_allowed":[],"desc":"pg_autoscale profiler","long_desc":"Determines the behavior of the autoscaler algorithm `scale-up` means that it starts out with minmum pgs and scales up when there is pressure, `scale-down` means starts out with full pgs and scales down when there is pressure ","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share perf counter metrics summed across the whole cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.197:0","nonce":1645829016}]},{"addrvec":[{"type":"v2","addr":"172.21.15.197:0","nonce":2022260216}]},{"addrvec":[{"type":"v2","addr":"172.21.15.197:0","nonce":1923538099}]},{"addrvec":[{"type":"v2","addr":"172.21.15.197:0","nonce":453132916}]}]}} 2021-09-02T07:27:59.612 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2021-09-02T07:27:59.613 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2021-09-02T07:27:59.613 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-09-02T07:27:59.892 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:27:59.893 INFO:teuthology.orchestra.run.smithi197.stdout:{"epoch":11,"fsid":"543d2dee-3303-4f4e-93e4-af35a71cd6ea","created":"2021-09-02T07:27:48.357746+0000","modified":"2021-09-02T07:27:59.317260+0000","last_up_change":"2021-09-02T07:27:56.159818+0000","last_in_change":"2021-09-02T07:27:49.285665+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-02T07:27:56.700049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-02T07:27:57.303114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"e0e9de1e-b652-4e2e-9dbf-8ab997b40365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6808","nonce":30570},{"type":"v1","addr":"172.21.15.197:6809","nonce":30570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6810","nonce":30570},{"type":"v1","addr":"172.21.15.197:6811","nonce":30570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6814","nonce":30570},{"type":"v1","addr":"172.21.15.197:6815","nonce":30570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6812","nonce":30570},{"type":"v1","addr":"172.21.15.197:6813","nonce":30570}]},"public_addr":"172.21.15.197:6809/30570","cluster_addr":"172.21.15.197:6811/30570","heartbeat_back_addr":"172.21.15.197:6815/30570","heartbeat_front_addr":"172.21.15.197:6813/30570","state":["exists","up"]},{"osd":1,"uuid":"56342390-0149-463b-bca1-27bdeb4d6d73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6800","nonce":30571},{"type":"v1","addr":"172.21.15.197:6801","nonce":30571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6802","nonce":30571},{"type":"v1","addr":"172.21.15.197:6803","nonce":30571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6806","nonce":30571},{"type":"v1","addr":"172.21.15.197:6807","nonce":30571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6804","nonce":30571},{"type":"v1","addr":"172.21.15.197:6805","nonce":30571}]},"public_addr":"172.21.15.197:6801/30571","cluster_addr":"172.21.15.197:6803/30571","heartbeat_back_addr":"172.21.15.197:6807/30571","heartbeat_front_addr":"172.21.15.197:6805/30571","state":["exists","up"]},{"osd":2,"uuid":"b1cdb47e-f589-4df4-b22d-585e435c01f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6816","nonce":30572},{"type":"v1","addr":"172.21.15.197:6817","nonce":30572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6818","nonce":30572},{"type":"v1","addr":"172.21.15.197:6819","nonce":30572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6822","nonce":30572},{"type":"v1","addr":"172.21.15.197:6823","nonce":30572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6820","nonce":30572},{"type":"v1","addr":"172.21.15.197:6821","nonce":30572}]},"public_addr":"172.21.15.197:6817/30572","cluster_addr":"172.21.15.197:6819/30572","heartbeat_back_addr":"172.21.15.197:6823/30572","heartbeat_front_addr":"172.21.15.197:6821/30572","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-02T07:27:53.664089+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-02T07:27:53.706923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-02T07:27:53.738200+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-02T07:27:59.905 INFO:tasks.ceph.ceph_manager.c2:all up! 2021-09-02T07:27:59.906 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-09-02T07:28:00.185 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:28:00.186 INFO:teuthology.orchestra.run.smithi197.stdout:{"epoch":11,"fsid":"543d2dee-3303-4f4e-93e4-af35a71cd6ea","created":"2021-09-02T07:27:48.357746+0000","modified":"2021-09-02T07:27:59.317260+0000","last_up_change":"2021-09-02T07:27:56.159818+0000","last_in_change":"2021-09-02T07:27:49.285665+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-02T07:27:56.700049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-02T07:27:57.303114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"e0e9de1e-b652-4e2e-9dbf-8ab997b40365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6808","nonce":30570},{"type":"v1","addr":"172.21.15.197:6809","nonce":30570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6810","nonce":30570},{"type":"v1","addr":"172.21.15.197:6811","nonce":30570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6814","nonce":30570},{"type":"v1","addr":"172.21.15.197:6815","nonce":30570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6812","nonce":30570},{"type":"v1","addr":"172.21.15.197:6813","nonce":30570}]},"public_addr":"172.21.15.197:6809/30570","cluster_addr":"172.21.15.197:6811/30570","heartbeat_back_addr":"172.21.15.197:6815/30570","heartbeat_front_addr":"172.21.15.197:6813/30570","state":["exists","up"]},{"osd":1,"uuid":"56342390-0149-463b-bca1-27bdeb4d6d73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6800","nonce":30571},{"type":"v1","addr":"172.21.15.197:6801","nonce":30571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6802","nonce":30571},{"type":"v1","addr":"172.21.15.197:6803","nonce":30571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6806","nonce":30571},{"type":"v1","addr":"172.21.15.197:6807","nonce":30571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6804","nonce":30571},{"type":"v1","addr":"172.21.15.197:6805","nonce":30571}]},"public_addr":"172.21.15.197:6801/30571","cluster_addr":"172.21.15.197:6803/30571","heartbeat_back_addr":"172.21.15.197:6807/30571","heartbeat_front_addr":"172.21.15.197:6805/30571","state":["exists","up"]},{"osd":2,"uuid":"b1cdb47e-f589-4df4-b22d-585e435c01f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6816","nonce":30572},{"type":"v1","addr":"172.21.15.197:6817","nonce":30572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6818","nonce":30572},{"type":"v1","addr":"172.21.15.197:6819","nonce":30572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6822","nonce":30572},{"type":"v1","addr":"172.21.15.197:6823","nonce":30572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.197:6820","nonce":30572},{"type":"v1","addr":"172.21.15.197:6821","nonce":30572}]},"public_addr":"172.21.15.197:6817/30572","cluster_addr":"172.21.15.197:6819/30572","heartbeat_back_addr":"172.21.15.197:6823/30572","heartbeat_front_addr":"172.21.15.197:6821/30572","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-02T07:27:53.664089+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-02T07:27:53.706923+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-02T07:27:53.738200+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-02T07:28:00.199 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2021-09-02T07:28:00.342 INFO:teuthology.orchestra.run.smithi197.stdout:34359738371 2021-09-02T07:28:00.343 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2021-09-02T07:28:00.487 INFO:teuthology.orchestra.run.smithi197.stdout:34359738371 2021-09-02T07:28:00.487 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2021-09-02T07:28:00.630 INFO:teuthology.orchestra.run.smithi197.stdout:34359738372 2021-09-02T07:28:00.631 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-09-02T07:28:00.913 INFO:teuthology.orchestra.run.smithi197.stdout:34359738370 2021-09-02T07:28:00.927 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2021-09-02T07:28:01.928 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-09-02T07:28:02.223 INFO:teuthology.orchestra.run.smithi197.stdout:34359738371 2021-09-02T07:28:02.236 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.0 2021-09-02T07:28:02.236 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2021-09-02T07:28:02.518 INFO:teuthology.orchestra.run.smithi197.stdout:34359738371 2021-09-02T07:28:02.532 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.1 2021-09-02T07:28:02.532 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2021-09-02T07:28:02.819 INFO:teuthology.orchestra.run.smithi197.stdout:34359738372 2021-09-02T07:28:02.830 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.2 2021-09-02T07:28:02.830 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2021-09-02T07:28:02.831 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-09-02T07:28:03.106 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:28:03.107 INFO:teuthology.orchestra.run.smithi197.stderr:dumped all 2021-09-02T07:28:03.119 INFO:teuthology.orchestra.run.smithi197.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2021-09-02T07:28:01.276083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24552,"kb_used_data":1424,"kb_used_omap":0,"kb_used_meta":23104,"kb_avail":314548248,"statfs":{"total":322122547200,"available":322097405952,"internally_reserved":0,"allocated":1458176,"data_stored":1249530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23658496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-02T07:27:58.329486+0000","last_change":"2021-09-02T07:27:58.327262+0000","last_active":"2021-09-02T07:27:58.329486+0000","last_peered":"2021-09-02T07:27:58.329486+0000","last_clean":"2021-09-02T07:27:58.329486+0000","last_became_active":"2021-09-02T07:27:58.326983+0000","last_became_peered":"2021-09-02T07:27:58.326983+0000","last_unstale":"2021-09-02T07:27:58.329486+0000","last_undegraded":"2021-09-02T07:27:58.329486+0000","last_fullsized":"2021-09-02T07:27:58.329486+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_clean_scrub_stamp":"2021-09-02T07:27:57.310676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-02T07:27:58.329250+0000","last_change":"2021-09-02T07:27:58.326210+0000","last_active":"2021-09-02T07:27:58.329250+0000","last_peered":"2021-09-02T07:27:58.329250+0000","last_clean":"2021-09-02T07:27:58.329250+0000","last_became_active":"2021-09-02T07:27:58.325951+0000","last_became_peered":"2021-09-02T07:27:58.325951+0000","last_unstale":"2021-09-02T07:27:58.329250+0000","last_undegraded":"2021-09-02T07:27:58.329250+0000","last_fullsized":"2021-09-02T07:27:58.329250+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_clean_scrub_stamp":"2021-09-02T07:27:57.310676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-02T07:27:58.325416+0000","last_change":"2021-09-02T07:27:58.324891+0000","last_active":"2021-09-02T07:27:58.325416+0000","last_peered":"2021-09-02T07:27:58.325416+0000","last_clean":"2021-09-02T07:27:58.325416+0000","last_became_active":"2021-09-02T07:27:58.324423+0000","last_became_peered":"2021-09-02T07:27:58.324423+0000","last_unstale":"2021-09-02T07:27:58.325416+0000","last_undegraded":"2021-09-02T07:27:58.325416+0000","last_fullsized":"2021-09-02T07:27:58.325416+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_clean_scrub_stamp":"2021-09-02T07:27:57.310676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-02T07:27:58.329272+0000","last_change":"2021-09-02T07:27:58.326813+0000","last_active":"2021-09-02T07:27:58.329272+0000","last_peered":"2021-09-02T07:27:58.329272+0000","last_clean":"2021-09-02T07:27:58.329272+0000","last_became_active":"2021-09-02T07:27:58.326042+0000","last_became_peered":"2021-09-02T07:27:58.326042+0000","last_unstale":"2021-09-02T07:27:58.329272+0000","last_undegraded":"2021-09-02T07:27:58.329272+0000","last_fullsized":"2021-09-02T07:27:58.329272+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_clean_scrub_stamp":"2021-09-02T07:27:57.310676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-02T07:27:58.326328+0000","last_change":"2021-09-02T07:27:58.325394+0000","last_active":"2021-09-02T07:27:58.326328+0000","last_peered":"2021-09-02T07:27:58.326328+0000","last_clean":"2021-09-02T07:27:58.326328+0000","last_became_active":"2021-09-02T07:27:58.325092+0000","last_became_peered":"2021-09-02T07:27:58.325092+0000","last_unstale":"2021-09-02T07:27:58.326328+0000","last_undegraded":"2021-09-02T07:27:58.326328+0000","last_fullsized":"2021-09-02T07:27:58.326328+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_clean_scrub_stamp":"2021-09-02T07:27:57.310676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-02T07:27:58.782570+0000","last_change":"2021-09-02T07:27:58.324410+0000","last_active":"2021-09-02T07:27:58.782570+0000","last_peered":"2021-09-02T07:27:58.782570+0000","last_clean":"2021-09-02T07:27:58.782570+0000","last_became_active":"2021-09-02T07:27:58.324083+0000","last_became_peered":"2021-09-02T07:27:58.324083+0000","last_unstale":"2021-09-02T07:27:58.782570+0000","last_undegraded":"2021-09-02T07:27:58.782570+0000","last_fullsized":"2021-09-02T07:27:58.782570+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_clean_scrub_stamp":"2021-09-02T07:27:57.310676+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-02T07:27:58.329259+0000","last_change":"2021-09-02T07:27:58.327052+0000","last_active":"2021-09-02T07:27:58.329259+0000","last_peered":"2021-09-02T07:27:58.329259+0000","last_clean":"2021-09-02T07:27:58.329259+0000","last_became_active":"2021-09-02T07:27:58.326757+0000","last_became_peered":"2021-09-02T07:27:58.326757+0000","last_unstale":"2021-09-02T07:27:58.329259+0000","last_undegraded":"2021-09-02T07:27:58.329259+0000","last_fullsized":"2021-09-02T07:27:58.329259+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_clean_scrub_stamp":"2021-09-02T07:27:57.310676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-02T07:27:58.329241+0000","last_change":"2021-09-02T07:27:58.326483+0000","last_active":"2021-09-02T07:27:58.329241+0000","last_peered":"2021-09-02T07:27:58.329241+0000","last_clean":"2021-09-02T07:27:58.329241+0000","last_became_active":"2021-09-02T07:27:58.326199+0000","last_became_peered":"2021-09-02T07:27:58.326199+0000","last_unstale":"2021-09-02T07:27:58.329241+0000","last_undegraded":"2021-09-02T07:27:58.329241+0000","last_fullsized":"2021-09-02T07:27:58.329241+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_clean_scrub_stamp":"2021-09-02T07:27:57.310676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-02T07:27:58.325409+0000","last_change":"2021-09-02T07:27:58.324828+0000","last_active":"2021-09-02T07:27:58.325409+0000","last_peered":"2021-09-02T07:27:58.325409+0000","last_clean":"2021-09-02T07:27:58.325409+0000","last_became_active":"2021-09-02T07:27:58.324539+0000","last_became_peered":"2021-09-02T07:27:58.324539+0000","last_unstale":"2021-09-02T07:27:58.325409+0000","last_undegraded":"2021-09-02T07:27:58.325409+0000","last_fullsized":"2021-09-02T07:27:58.325409+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_clean_scrub_stamp":"2021-09-02T07:27:57.310676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8480,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":104849120,"statfs":{"total":107374182400,"available":107365498880,"internally_reserved":0,"allocated":679936,"data_stored":611669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7516,"kb_used_data":84,"kb_used_omap":0,"kb_used_meta":7424,"kb_avail":104850084,"statfs":{"total":107374182400,"available":107366486016,"internally_reserved":0,"allocated":86016,"data_stored":21301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7602176},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8556,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104849044,"statfs":{"total":107374182400,"available":107365421056,"internally_reserved":0,"allocated":692224,"data_stored":616560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-02T07:28:03.121 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-09-02T07:28:03.402 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:28:03.403 INFO:teuthology.orchestra.run.smithi197.stderr:dumped all 2021-09-02T07:28:03.414 INFO:teuthology.orchestra.run.smithi197.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2021-09-02T07:28:03.276475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24632,"kb_used_data":1440,"kb_used_omap":0,"kb_used_meta":23168,"kb_avail":314548168,"statfs":{"total":322122547200,"available":322097324032,"internally_reserved":0,"allocated":1474560,"data_stored":1257138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23724032},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000392"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-02T07:27:58.329486+0000","last_change":"2021-09-02T07:27:58.327262+0000","last_active":"2021-09-02T07:27:58.329486+0000","last_peered":"2021-09-02T07:27:58.329486+0000","last_clean":"2021-09-02T07:27:58.329486+0000","last_became_active":"2021-09-02T07:27:58.326983+0000","last_became_peered":"2021-09-02T07:27:58.326983+0000","last_unstale":"2021-09-02T07:27:58.329486+0000","last_undegraded":"2021-09-02T07:27:58.329486+0000","last_fullsized":"2021-09-02T07:27:58.329486+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_clean_scrub_stamp":"2021-09-02T07:27:57.310676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-02T07:27:58.329250+0000","last_change":"2021-09-02T07:27:58.326210+0000","last_active":"2021-09-02T07:27:58.329250+0000","last_peered":"2021-09-02T07:27:58.329250+0000","last_clean":"2021-09-02T07:27:58.329250+0000","last_became_active":"2021-09-02T07:27:58.325951+0000","last_became_peered":"2021-09-02T07:27:58.325951+0000","last_unstale":"2021-09-02T07:27:58.329250+0000","last_undegraded":"2021-09-02T07:27:58.329250+0000","last_fullsized":"2021-09-02T07:27:58.329250+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_clean_scrub_stamp":"2021-09-02T07:27:57.310676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-02T07:27:58.325416+0000","last_change":"2021-09-02T07:27:58.324891+0000","last_active":"2021-09-02T07:27:58.325416+0000","last_peered":"2021-09-02T07:27:58.325416+0000","last_clean":"2021-09-02T07:27:58.325416+0000","last_became_active":"2021-09-02T07:27:58.324423+0000","last_became_peered":"2021-09-02T07:27:58.324423+0000","last_unstale":"2021-09-02T07:27:58.325416+0000","last_undegraded":"2021-09-02T07:27:58.325416+0000","last_fullsized":"2021-09-02T07:27:58.325416+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_clean_scrub_stamp":"2021-09-02T07:27:57.310676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-02T07:27:58.329272+0000","last_change":"2021-09-02T07:27:58.326813+0000","last_active":"2021-09-02T07:27:58.329272+0000","last_peered":"2021-09-02T07:27:58.329272+0000","last_clean":"2021-09-02T07:27:58.329272+0000","last_became_active":"2021-09-02T07:27:58.326042+0000","last_became_peered":"2021-09-02T07:27:58.326042+0000","last_unstale":"2021-09-02T07:27:58.329272+0000","last_undegraded":"2021-09-02T07:27:58.329272+0000","last_fullsized":"2021-09-02T07:27:58.329272+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_clean_scrub_stamp":"2021-09-02T07:27:57.310676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-02T07:27:58.326328+0000","last_change":"2021-09-02T07:27:58.325394+0000","last_active":"2021-09-02T07:27:58.326328+0000","last_peered":"2021-09-02T07:27:58.326328+0000","last_clean":"2021-09-02T07:27:58.326328+0000","last_became_active":"2021-09-02T07:27:58.325092+0000","last_became_peered":"2021-09-02T07:27:58.325092+0000","last_unstale":"2021-09-02T07:27:58.326328+0000","last_undegraded":"2021-09-02T07:27:58.326328+0000","last_fullsized":"2021-09-02T07:27:58.326328+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_clean_scrub_stamp":"2021-09-02T07:27:57.310676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-02T07:27:58.782570+0000","last_change":"2021-09-02T07:27:58.324410+0000","last_active":"2021-09-02T07:27:58.782570+0000","last_peered":"2021-09-02T07:27:58.782570+0000","last_clean":"2021-09-02T07:27:58.782570+0000","last_became_active":"2021-09-02T07:27:58.324083+0000","last_became_peered":"2021-09-02T07:27:58.324083+0000","last_unstale":"2021-09-02T07:27:58.782570+0000","last_undegraded":"2021-09-02T07:27:58.782570+0000","last_fullsized":"2021-09-02T07:27:58.782570+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_clean_scrub_stamp":"2021-09-02T07:27:57.310676+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-02T07:27:58.329259+0000","last_change":"2021-09-02T07:27:58.327052+0000","last_active":"2021-09-02T07:27:58.329259+0000","last_peered":"2021-09-02T07:27:58.329259+0000","last_clean":"2021-09-02T07:27:58.329259+0000","last_became_active":"2021-09-02T07:27:58.326757+0000","last_became_peered":"2021-09-02T07:27:58.326757+0000","last_unstale":"2021-09-02T07:27:58.329259+0000","last_undegraded":"2021-09-02T07:27:58.329259+0000","last_fullsized":"2021-09-02T07:27:58.329259+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_clean_scrub_stamp":"2021-09-02T07:27:57.310676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-02T07:27:58.329241+0000","last_change":"2021-09-02T07:27:58.326483+0000","last_active":"2021-09-02T07:27:58.329241+0000","last_peered":"2021-09-02T07:27:58.329241+0000","last_clean":"2021-09-02T07:27:58.329241+0000","last_became_active":"2021-09-02T07:27:58.326199+0000","last_became_peered":"2021-09-02T07:27:58.326199+0000","last_unstale":"2021-09-02T07:27:58.329241+0000","last_undegraded":"2021-09-02T07:27:58.329241+0000","last_fullsized":"2021-09-02T07:27:58.329241+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_clean_scrub_stamp":"2021-09-02T07:27:57.310676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-02T07:27:58.325409+0000","last_change":"2021-09-02T07:27:58.324828+0000","last_active":"2021-09-02T07:27:58.325409+0000","last_peered":"2021-09-02T07:27:58.325409+0000","last_clean":"2021-09-02T07:27:58.325409+0000","last_became_active":"2021-09-02T07:27:58.324539+0000","last_became_peered":"2021-09-02T07:27:58.324539+0000","last_unstale":"2021-09-02T07:27:58.325409+0000","last_undegraded":"2021-09-02T07:27:58.325409+0000","last_fullsized":"2021-09-02T07:27:58.325409+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-02T07:27:57.310676+0000","last_clean_scrub_stamp":"2021-09-02T07:27:57.310676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8488,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":104849112,"statfs":{"total":107374182400,"available":107365490688,"internally_reserved":0,"allocated":688128,"data_stored":615473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7588,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850012,"statfs":{"total":107374182400,"available":107366412288,"internally_reserved":0,"allocated":94208,"data_stored":25105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8556,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104849044,"statfs":{"total":107374182400,"available":107365421056,"internally_reserved":0,"allocated":692224,"data_stored":616560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-02T07:28:03.415 INFO:tasks.ceph.ceph_manager.c2:clean! 2021-09-02T07:28:03.416 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2021-09-02T07:28:03.416 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2021-09-02T07:28:03.417 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2021-09-02T07:28:03.757 INFO:teuthology.orchestra.run.smithi197.stdout: 2021-09-02T07:28:03.758 INFO:teuthology.orchestra.run.smithi197.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-09-02T07:28:03.770 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2021-09-02T07:28:03.770 INFO:teuthology.run_tasks:Running task rgw... 2021-09-02T07:28:03.785 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2021-09-02T07:28:03.785 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2021-09-02T07:28:03.786 INFO:tasks.rgw:Creating data pools 2021-09-02T07:28:03.786 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2021-09-02T07:28:03.787 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-09-02T07:28:04.719 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'default.rgw.buckets.data' created 2021-09-02T07:28:04.736 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-09-02T07:28:05.749 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-02T07:28:05.766 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-09-02T07:28:06.741 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'default.rgw.buckets.index' created 2021-09-02T07:28:06.764 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-09-02T07:28:07.757 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-02T07:28:07.779 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2021-09-02T07:28:07.779 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-09-02T07:28:08.061 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'default.rgw.buckets.data' already exists 2021-09-02T07:28:08.071 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-09-02T07:28:08.753 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-02T07:28:08.764 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-09-02T07:28:09.045 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'default.rgw.buckets.index' already exists 2021-09-02T07:28:09.055 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-09-02T07:28:09.759 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-02T07:28:09.770 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2021-09-02T07:28:09.771 DEBUG:teuthology.orchestra.run.smithi197:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-09-02T07:28:10.363 INFO:teuthology.orchestra.run.smithi197.stderr:pool 'default.rgw.buckets.data' created 2021-09-02T07:28:10.378 DEBUG:teuthology.orchestra.run.smithi197:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-09-02T07:28:11.375 INFO:teuthology.orchestra.run.smithi197.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-02T07:28:11.399 DEBUG:teuthology.orchestra.run.smithi197:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-09-02T07:28:12.374 INFO:teuthology.orchestra.run.smithi197.stderr:pool 'default.rgw.buckets.index' created 2021-09-02T07:28:12.385 DEBUG:teuthology.orchestra.run.smithi197:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-09-02T07:28:13.382 INFO:teuthology.orchestra.run.smithi197.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-02T07:28:13.401 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2021-09-02T07:28:13.402 DEBUG:teuthology.orchestra.run.smithi197:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-09-02T07:28:13.689 INFO:teuthology.orchestra.run.smithi197.stderr:pool 'default.rgw.buckets.data' already exists 2021-09-02T07:28:13.700 DEBUG:teuthology.orchestra.run.smithi197:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-09-02T07:28:14.372 INFO:teuthology.orchestra.run.smithi197.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-02T07:28:14.383 DEBUG:teuthology.orchestra.run.smithi197:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-09-02T07:28:14.671 INFO:teuthology.orchestra.run.smithi197.stderr:pool 'default.rgw.buckets.index' already exists 2021-09-02T07:28:14.682 DEBUG:teuthology.orchestra.run.smithi197:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-09-02T07:28:15.377 INFO:teuthology.orchestra.run.smithi197.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-02T07:28:15.389 DEBUG:tasks.rgw:Pools created 2021-09-02T07:28:15.390 INFO:tasks.rgw:Configuring compression type = random 2021-09-02T07:28:15.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2021-09-02T07:28:15.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2021-09-02T07:28:15.391 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2021-09-02T07:28:15.507 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:28:15.507 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:28:15.513 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:15.515+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.784 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.786+0000 7f6391b4c380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:16.785 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.786+0000 7f6391b4c380 20 realm 2021-09-02T07:28:16.785 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.786+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.787 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.787+0000 7f6391b4c380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:16.787 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.787+0000 7f6391b4c380 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-02T07:28:16.787 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.787+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.787 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.787+0000 7f6391b4c380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:16.788 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.787+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.788 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.788+0000 7f6391b4c380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:16.788 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.788+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.789 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.789+0000 7f6391b4c380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:16.789 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.789+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.789 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.789+0000 7f6391b4c380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:16.790 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.789+0000 7f6391b4c380 10 Creating default zonegroup 2021-09-02T07:28:16.790 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.789+0000 7f6391b4c380 20 get_system_obj_state: rctx=0x7ffef355f760 obj=:.pools.avail state=0x55d06e379d70 s->prefetch_data=0 2021-09-02T07:28:16.790 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.790+0000 7f6391b4c380 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-09-02T07:28:16.808 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.809+0000 7f6391b4c380 10 WARNING: store->list_zones() returned r=-2 2021-09-02T07:28:16.809 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.809+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.809 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.810+0000 7f6391b4c380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:16.811 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.813+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.812 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.813+0000 7f6391b4c380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:16.812 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.813+0000 7f6391b4c380 10 could not read realm id: (2) No such file or directory 2021-09-02T07:28:16.812 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.813+0000 7f6391b4c380 10 WARNING: failed to set zone as default, r=-22 2021-09-02T07:28:16.812 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.813+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.812 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.813+0000 7f6391b4c380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:16.814 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.816+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.815 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.817+0000 7f6391b4c380 20 rados_obj.operate() r=0 bl.length=873 2021-09-02T07:28:16.815 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.817+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.816 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.817+0000 7f6391b4c380 20 rados_obj.operate() r=0 bl.length=358 2021-09-02T07:28:16.817 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.817+0000 7f6391b4c380 10 Using default name default 2021-09-02T07:28:16.817 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.817+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.817 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.818+0000 7f6391b4c380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:16.817 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.818+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.818 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.818+0000 7f6391b4c380 20 rados_obj.operate() r=0 bl.length=873 2021-09-02T07:28:16.818 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.818+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.818 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.818+0000 7f6391b4c380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:16.818 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.818+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.819 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.819+0000 7f6391b4c380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:16.819 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.819+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.820 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.819+0000 7f6391b4c380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:16.828 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.830+0000 7f6391b4c380 20 RGWRados::pool_iterate: got zonegroup_info.c7bb63f3-7908-4e63-9096-6856e64619e6 2021-09-02T07:28:16.829 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.830+0000 7f6391b4c380 20 RGWRados::pool_iterate: got zone_info.71986023-b31f-431b-9e99-0a21d50112a1 2021-09-02T07:28:16.829 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.830+0000 7f6391b4c380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-02T07:28:16.829 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.830+0000 7f6391b4c380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-02T07:28:16.829 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.830+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.830 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.830+0000 7f6391b4c380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:16.830 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.830+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.831 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.831+0000 7f6391b4c380 20 rados_obj.operate() r=0 bl.length=873 2021-09-02T07:28:16.832 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.831+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.832 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.831+0000 7f6391b4c380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:16.832 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.831+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.832 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.831+0000 7f6391b4c380 20 rados_obj.operate() r=0 bl.length=358 2021-09-02T07:28:16.833 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.831+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.833 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.832+0000 7f6391b4c380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:16.833 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.832+0000 7f6391b4c380 20 zone default found 2021-09-02T07:28:16.833 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.832+0000 7f6391b4c380 1 Realm: () 2021-09-02T07:28:16.833 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.832+0000 7f6391b4c380 1 ZoneGroup: default (c7bb63f3-7908-4e63-9096-6856e64619e6) 2021-09-02T07:28:16.834 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.832+0000 7f6391b4c380 1 Zone: default (71986023-b31f-431b-9e99-0a21d50112a1) 2021-09-02T07:28:16.834 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.832+0000 7f6391b4c380 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-02T07:28:16.834 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.832+0000 7f6391b4c380 20 zonegroup default 2021-09-02T07:28:16.834 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.832+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:16.835 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.832+0000 7f6391b4c380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:16.835 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:16.832+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:18.796 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:18.797+0000 7f6391b4c380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:18.797 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:18.797+0000 7f6391b4c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:18.798 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:18.799+0000 7f6391b4c380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:18.798 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:18.799+0000 7f6391b4c380 20 started sync module instance, tier type = 2021-09-02T07:28:18.801 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:18.799+0000 7f6391b4c380 20 started zone id=71986023-b31f-431b-9e99-0a21d50112a1 (name=default) with tier type = 2021-09-02T07:28:20.848 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.850+0000 7f6391b4c380 20 add_watcher() i=0 2021-09-02T07:28:20.848 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.850+0000 7f6391b4c380 20 add_watcher() i=1 2021-09-02T07:28:20.849 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.850+0000 7f6391b4c380 20 add_watcher() i=2 2021-09-02T07:28:20.849 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.850+0000 7f6391b4c380 20 add_watcher() i=3 2021-09-02T07:28:20.849 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.850+0000 7f6391b4c380 20 add_watcher() i=4 2021-09-02T07:28:20.849 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.850+0000 7f6391b4c380 20 add_watcher() i=5 2021-09-02T07:28:20.850 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.850+0000 7f6391b4c380 20 add_watcher() i=6 2021-09-02T07:28:20.851 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.852+0000 7f6391b4c380 20 add_watcher() i=7 2021-09-02T07:28:20.852 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.852+0000 7f6391b4c380 2 all 8 watchers are set, enabling cache 2021-09-02T07:28:20.852 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.852+0000 7f6391b4c380 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2021-09-02T07:28:20.852 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.852+0000 7f6391b4c380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2021-09-02T07:28:20.852 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.852+0000 7f6391b4c380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-02T07:28:20.853 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.853+0000 7f6391b4c380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2021-09-02T07:28:20.853 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.853+0000 7f6391b4c380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2021-09-02T07:28:20.855 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.853+0000 7f6391b4c380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-02T07:28:20.855 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.853+0000 7f6391b4c380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2021-09-02T07:28:20.855 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.853+0000 7f6391b4c380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2021-09-02T07:28:20.855 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.853+0000 7f6391b4c380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-02T07:28:20.856 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.853+0000 7f6391b4c380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2021-09-02T07:28:20.856 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.853+0000 7f6391b4c380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2021-09-02T07:28:20.856 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.853+0000 7f6391b4c380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-02T07:28:20.856 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.854+0000 7f6391b4c380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2021-09-02T07:28:20.857 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.854+0000 7f6391b4c380 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2021-09-02T07:28:20.857 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.855+0000 7f6391b4c380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-02T07:28:20.863 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.864+0000 7f6355fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-02T07:28:20.863 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.864+0000 7f6391b4c380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-02T07:28:20.863 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:20.864+0000 7f6391b4c380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-02T07:28:24.796 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.795+0000 7f6391b4c380 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-09-02T07:28:24.796 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.795+0000 7f6391b4c380 5 note: GC not initialized 2021-09-02T07:28:24.796 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.797+0000 7f6391b4c380 20 init_complete bucket index max shards: 11 2021-09-02T07:28:24.797 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.797+0000 7f6391b4c380 10 rgw notify: Started notification manager with: 1 workers 2021-09-02T07:28:24.797 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.797+0000 7f62c4ff9700 20 reqs_thread_entry: start 2021-09-02T07:28:24.797 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.797+0000 7f62b67fc700 20 reqs_thread_entry: start 2021-09-02T07:28:24.798 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.797+0000 7f62b4ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Sep 2 07:28:55 2021 2021-09-02T07:28:24.798 INFO:teuthology.orchestra.run.smithi175.stderr: 2021-09-02T07:28:24.798 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.797+0000 7f629ffff700 20 reqs_thread_entry: start 2021-09-02T07:28:24.822 INFO:teuthology.orchestra.run.smithi175.stdout:[] 2021-09-02T07:28:24.823 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.824+0000 7f6391b4c380 20 remove_watcher() i=0 2021-09-02T07:28:24.824 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.824+0000 7f6391b4c380 2 removed watcher, disabling cache 2021-09-02T07:28:24.824 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.825+0000 7f6391b4c380 20 remove_watcher() i=1 2021-09-02T07:28:24.826 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.828+0000 7f6391b4c380 20 remove_watcher() i=2 2021-09-02T07:28:24.831 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.833+0000 7f6391b4c380 20 remove_watcher() i=3 2021-09-02T07:28:24.832 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.834+0000 7f6391b4c380 20 remove_watcher() i=4 2021-09-02T07:28:24.834 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.836+0000 7f6391b4c380 20 remove_watcher() i=5 2021-09-02T07:28:24.835 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.837+0000 7f6391b4c380 20 remove_watcher() i=6 2021-09-02T07:28:24.841 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.843+0000 7f6391b4c380 20 remove_watcher() i=7 2021-09-02T07:28:24.842 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.843+0000 7f6355fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-02T07:28:24.847 DEBUG:tasks.util.rgw: json result: [] 2021-09-02T07:28:24.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-02T07:28:24.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-02T07:28:24.850 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-02T07:28:24.895 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:28:24.896 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:28:24.901 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.903+0000 7f0d6c97c380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-02T07:28:24.901 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.903+0000 7f0d6c97c380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-02T07:28:24.905 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.906+0000 7f0ce97fa700 20 reqs_thread_entry: start 2021-09-02T07:28:24.905 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.906+0000 7f0d6c97c380 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-02T07:28:24.905 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.906+0000 7f0d6c97c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:24.906 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.907+0000 7f0d6c97c380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:24.907 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.907+0000 7f0d6c97c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:24.907 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.908+0000 7f0d6c97c380 20 rados_obj.operate() r=0 bl.length=873 2021-09-02T07:28:24.917 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.919+0000 7f0d6c97c380 20 RGWRados::pool_iterate: got zonegroup_info.c7bb63f3-7908-4e63-9096-6856e64619e6 2021-09-02T07:28:24.917 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.919+0000 7f0d6c97c380 20 RGWRados::pool_iterate: got zone_info.71986023-b31f-431b-9e99-0a21d50112a1 2021-09-02T07:28:24.918 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.919+0000 7f0d6c97c380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-02T07:28:24.918 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.919+0000 7f0d6c97c380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-02T07:28:24.918 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.919+0000 7f0d6c97c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:24.918 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.919+0000 7f0d6c97c380 20 rados_obj.operate() r=0 bl.length=873 2021-09-02T07:28:24.919 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.919+0000 7f0d6c97c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:24.919 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.919+0000 7f0d6c97c380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:24.919 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.919+0000 7f0d6c97c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:24.920 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.920+0000 7f0d6c97c380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:24.920 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.920+0000 7f0d6c97c380 20 rados->read ofs=0 len=0 2021-09-02T07:28:24.920 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.920+0000 7f0d6c97c380 20 rados_obj.operate() r=0 bl.length=358 2021-09-02T07:28:24.922 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:28:24.922 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "71986023-b31f-431b-9e99-0a21d50112a1", 2021-09-02T07:28:24.923 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default", 2021-09-02T07:28:24.923 INFO:teuthology.orchestra.run.smithi175.stdout: "domain_root": "default.rgw.meta:root", 2021-09-02T07:28:24.923 INFO:teuthology.orchestra.run.smithi175.stdout: "control_pool": "default.rgw.control", 2021-09-02T07:28:24.923 INFO:teuthology.orchestra.run.smithi175.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-02T07:28:24.924 INFO:teuthology.orchestra.run.smithi175.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-02T07:28:24.924 INFO:teuthology.orchestra.run.smithi175.stdout: "log_pool": "default.rgw.log", 2021-09-02T07:28:24.924 INFO:teuthology.orchestra.run.smithi175.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-02T07:28:24.924 INFO:teuthology.orchestra.run.smithi175.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-02T07:28:24.925 INFO:teuthology.orchestra.run.smithi175.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-02T07:28:24.925 INFO:teuthology.orchestra.run.smithi175.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-02T07:28:24.925 INFO:teuthology.orchestra.run.smithi175.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-02T07:28:24.925 INFO:teuthology.orchestra.run.smithi175.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-02T07:28:24.925 INFO:teuthology.orchestra.run.smithi175.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-02T07:28:24.926 INFO:teuthology.orchestra.run.smithi175.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-02T07:28:24.926 INFO:teuthology.orchestra.run.smithi175.stdout: "otp_pool": "default.rgw.otp", 2021-09-02T07:28:24.926 INFO:teuthology.orchestra.run.smithi175.stdout: "system_key": { 2021-09-02T07:28:24.926 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "", 2021-09-02T07:28:24.927 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "" 2021-09-02T07:28:24.927 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:28:24.927 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_pools": [ 2021-09-02T07:28:24.927 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:28:24.928 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "default-placement", 2021-09-02T07:28:24.928 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:28:24.928 INFO:teuthology.orchestra.run.smithi175.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-02T07:28:24.929 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": { 2021-09-02T07:28:24.930 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD": { 2021-09-02T07:28:24.930 INFO:teuthology.orchestra.run.smithi175.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-02T07:28:24.930 INFO:teuthology.orchestra.run.smithi175.stdout: "compression_type": "random" 2021-09-02T07:28:24.931 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:28:24.931 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:28:24.931 INFO:teuthology.orchestra.run.smithi175.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-02T07:28:24.931 INFO:teuthology.orchestra.run.smithi175.stdout: "index_type": 0 2021-09-02T07:28:24.932 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:28:24.932 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:28:24.932 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:28:24.932 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "", 2021-09-02T07:28:24.932 INFO:teuthology.orchestra.run.smithi175.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-02T07:28:24.933 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:28:24.935 DEBUG:tasks.util.rgw: json result: {'id': '71986023-b31f-431b-9e99-0a21d50112a1', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-02T07:28:24.935 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2021-09-02T07:28:24.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2021-09-02T07:28:24.936 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2021-09-02T07:28:24.974 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:28:24.974 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:28:24.978 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.980+0000 7fbab25ef380 20 rados->read ofs=0 len=0 2021-09-02T07:28:24.980 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.982+0000 7fbab25ef380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:24.980 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.982+0000 7fbab25ef380 20 realm 2021-09-02T07:28:24.981 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.982+0000 7fbab25ef380 20 rados->read ofs=0 len=0 2021-09-02T07:28:24.981 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.982+0000 7fbab25ef380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:24.981 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.982+0000 7fbab25ef380 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-02T07:28:24.981 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.982+0000 7fbab25ef380 20 rados->read ofs=0 len=0 2021-09-02T07:28:24.981 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.982+0000 7fbab25ef380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:24.982 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.982+0000 7fbab25ef380 20 rados->read ofs=0 len=0 2021-09-02T07:28:24.983 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.982+0000 7fbab25ef380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:24.983 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.982+0000 7fbab25ef380 20 rados->read ofs=0 len=0 2021-09-02T07:28:24.983 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.983+0000 7fbab25ef380 20 rados_obj.operate() r=0 bl.length=889 2021-09-02T07:28:24.983 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.983+0000 7fbab25ef380 20 searching for the correct realm 2021-09-02T07:28:24.991 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.993+0000 7fbab25ef380 20 RGWRados::pool_iterate: got zonegroup_info.c7bb63f3-7908-4e63-9096-6856e64619e6 2021-09-02T07:28:24.992 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.993+0000 7fbab25ef380 20 RGWRados::pool_iterate: got zone_info.71986023-b31f-431b-9e99-0a21d50112a1 2021-09-02T07:28:24.992 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.993+0000 7fbab25ef380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-02T07:28:24.992 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.993+0000 7fbab25ef380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-02T07:28:24.992 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.993+0000 7fbab25ef380 20 rados->read ofs=0 len=0 2021-09-02T07:28:24.992 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.993+0000 7fbab25ef380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:24.993 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.993+0000 7fbab25ef380 20 rados->read ofs=0 len=0 2021-09-02T07:28:24.993 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.993+0000 7fbab25ef380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:24.993 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.993+0000 7fbab25ef380 20 rados->read ofs=0 len=0 2021-09-02T07:28:24.994 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.994+0000 7fbab25ef380 20 rados_obj.operate() r=0 bl.length=358 2021-09-02T07:28:24.995 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.994+0000 7fbab25ef380 20 rados->read ofs=0 len=0 2021-09-02T07:28:24.995 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.994+0000 7fbab25ef380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:24.995 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.994+0000 7fbab25ef380 20 rados->read ofs=0 len=0 2021-09-02T07:28:24.995 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.994+0000 7fbab25ef380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:24.995 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.994+0000 7fbab25ef380 20 rados->read ofs=0 len=0 2021-09-02T07:28:24.996 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:24.995+0000 7fbab25ef380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:25.002 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.004+0000 7fbab25ef380 20 RGWRados::pool_iterate: got zonegroup_info.c7bb63f3-7908-4e63-9096-6856e64619e6 2021-09-02T07:28:25.003 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.004+0000 7fbab25ef380 20 RGWRados::pool_iterate: got zone_info.71986023-b31f-431b-9e99-0a21d50112a1 2021-09-02T07:28:25.003 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.004+0000 7fbab25ef380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-02T07:28:25.003 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.004+0000 7fbab25ef380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-02T07:28:25.003 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.004+0000 7fbab25ef380 20 rados->read ofs=0 len=0 2021-09-02T07:28:25.004 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.004+0000 7fbab25ef380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:25.004 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.004+0000 7fbab25ef380 20 rados->read ofs=0 len=0 2021-09-02T07:28:25.004 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.004+0000 7fbab25ef380 20 rados_obj.operate() r=0 bl.length=889 2021-09-02T07:28:25.004 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.004+0000 7fbab25ef380 20 rados->read ofs=0 len=0 2021-09-02T07:28:25.006 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.005+0000 7fbab25ef380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:25.007 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.005+0000 7fbab25ef380 20 rados->read ofs=0 len=0 2021-09-02T07:28:25.007 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.005+0000 7fbab25ef380 20 rados_obj.operate() r=0 bl.length=358 2021-09-02T07:28:25.007 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.005+0000 7fbab25ef380 20 rados->read ofs=0 len=0 2021-09-02T07:28:25.007 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.005+0000 7fbab25ef380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:25.007 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.005+0000 7fbab25ef380 20 zone default found 2021-09-02T07:28:25.008 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.005+0000 7fbab25ef380 1 Realm: () 2021-09-02T07:28:25.008 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.005+0000 7fbab25ef380 1 ZoneGroup: default (c7bb63f3-7908-4e63-9096-6856e64619e6) 2021-09-02T07:28:25.008 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.005+0000 7fbab25ef380 1 Zone: default (71986023-b31f-431b-9e99-0a21d50112a1) 2021-09-02T07:28:25.008 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.005+0000 7fbab25ef380 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-02T07:28:25.009 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.005+0000 7fbab25ef380 20 zonegroup default 2021-09-02T07:28:25.009 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.005+0000 7fbab25ef380 20 rados->read ofs=0 len=0 2021-09-02T07:28:25.009 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.006+0000 7fbab25ef380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:25.009 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.006+0000 7fbab25ef380 20 rados->read ofs=0 len=0 2021-09-02T07:28:25.009 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.006+0000 7fbab25ef380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:25.010 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.006+0000 7fbab25ef380 20 rados->read ofs=0 len=0 2021-09-02T07:28:25.010 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.006+0000 7fbab25ef380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:25.010 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.006+0000 7fbab25ef380 20 started sync module instance, tier type = 2021-09-02T07:28:25.010 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.006+0000 7fbab25ef380 20 started zone id=71986023-b31f-431b-9e99-0a21d50112a1 (name=default) with tier type = 2021-09-02T07:28:25.038 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.040+0000 7fbab25ef380 20 add_watcher() i=0 2021-09-02T07:28:25.039 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.040+0000 7fbab25ef380 20 add_watcher() i=1 2021-09-02T07:28:25.039 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.040+0000 7fbab25ef380 20 add_watcher() i=2 2021-09-02T07:28:25.039 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.040+0000 7fbab25ef380 20 add_watcher() i=3 2021-09-02T07:28:25.039 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.040+0000 7fbab25ef380 20 add_watcher() i=4 2021-09-02T07:28:25.040 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.040+0000 7fbab25ef380 20 add_watcher() i=5 2021-09-02T07:28:25.040 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.040+0000 7fbab25ef380 20 add_watcher() i=6 2021-09-02T07:28:25.040 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.041+0000 7fbab25ef380 20 add_watcher() i=7 2021-09-02T07:28:25.041 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.041+0000 7fbab25ef380 2 all 8 watchers are set, enabling cache 2021-09-02T07:28:25.042 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.043+0000 7fba767e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-02T07:28:25.042 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.043+0000 7fbab25ef380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-02T07:28:25.042 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.043+0000 7fbab25ef380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-02T07:28:25.045 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.047+0000 7fbab25ef380 5 note: GC not initialized 2021-09-02T07:28:25.047 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.048+0000 7fb9e57fa700 20 reqs_thread_entry: start 2021-09-02T07:28:25.047 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.048+0000 7fb9deffd700 20 reqs_thread_entry: start 2021-09-02T07:28:25.047 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.048+0000 7fbab25ef380 20 init_complete bucket index max shards: 11 2021-09-02T07:28:25.047 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.048+0000 7fbab25ef380 10 rgw notify: Started notification manager with: 1 workers 2021-09-02T07:28:25.047 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.048+0000 7fb9dd7fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Sep 2 07:28:55 2021 2021-09-02T07:28:25.048 INFO:teuthology.orchestra.run.smithi175.stderr: 2021-09-02T07:28:25.048 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.048+0000 7fb9dcff9700 20 reqs_thread_entry: start 2021-09-02T07:28:25.057 INFO:teuthology.orchestra.run.smithi175.stdout:[] 2021-09-02T07:28:25.058 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.059+0000 7fbab25ef380 20 remove_watcher() i=0 2021-09-02T07:28:25.059 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.059+0000 7fbab25ef380 2 removed watcher, disabling cache 2021-09-02T07:28:25.059 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.060+0000 7fbab25ef380 20 remove_watcher() i=1 2021-09-02T07:28:25.059 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.061+0000 7fbab25ef380 20 remove_watcher() i=2 2021-09-02T07:28:25.061 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.063+0000 7fbab25ef380 20 remove_watcher() i=3 2021-09-02T07:28:25.062 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.064+0000 7fbab25ef380 20 remove_watcher() i=4 2021-09-02T07:28:25.063 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.065+0000 7fbab25ef380 20 remove_watcher() i=5 2021-09-02T07:28:25.064 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.066+0000 7fbab25ef380 20 remove_watcher() i=6 2021-09-02T07:28:25.065 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.067+0000 7fbab25ef380 20 remove_watcher() i=7 2021-09-02T07:28:25.066 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.068+0000 7fba767e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-02T07:28:25.075 DEBUG:tasks.util.rgw: json result: [] 2021-09-02T07:28:25.075 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-02T07:28:25.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-02T07:28:25.076 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-02T07:28:25.157 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:28:25.158 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:28:25.162 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.164+0000 7f4ee9525380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-02T07:28:25.164 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.164+0000 7f4ee9525380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-02T07:28:25.166 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.167+0000 7f4e25ffb700 20 reqs_thread_entry: start 2021-09-02T07:28:25.166 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.167+0000 7f4ee9525380 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-02T07:28:25.166 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.168+0000 7f4ee9525380 20 rados->read ofs=0 len=0 2021-09-02T07:28:25.167 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.168+0000 7f4ee9525380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:25.167 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.168+0000 7f4ee9525380 20 rados->read ofs=0 len=0 2021-09-02T07:28:25.168 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.169+0000 7f4ee9525380 20 rados_obj.operate() r=0 bl.length=889 2021-09-02T07:28:25.179 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.180+0000 7f4ee9525380 20 RGWRados::pool_iterate: got zonegroup_info.c7bb63f3-7908-4e63-9096-6856e64619e6 2021-09-02T07:28:25.179 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.180+0000 7f4ee9525380 20 RGWRados::pool_iterate: got zone_info.71986023-b31f-431b-9e99-0a21d50112a1 2021-09-02T07:28:25.179 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.180+0000 7f4ee9525380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-02T07:28:25.180 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.180+0000 7f4ee9525380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-02T07:28:25.180 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.180+0000 7f4ee9525380 20 rados->read ofs=0 len=0 2021-09-02T07:28:25.180 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.181+0000 7f4ee9525380 20 rados_obj.operate() r=0 bl.length=889 2021-09-02T07:28:25.180 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.181+0000 7f4ee9525380 20 rados->read ofs=0 len=0 2021-09-02T07:28:25.181 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.181+0000 7f4ee9525380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:25.181 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.181+0000 7f4ee9525380 20 rados->read ofs=0 len=0 2021-09-02T07:28:25.182 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.181+0000 7f4ee9525380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:25.182 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.181+0000 7f4ee9525380 20 rados->read ofs=0 len=0 2021-09-02T07:28:25.182 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:25.182+0000 7f4ee9525380 20 rados_obj.operate() r=0 bl.length=358 2021-09-02T07:28:25.184 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:28:25.184 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "71986023-b31f-431b-9e99-0a21d50112a1", 2021-09-02T07:28:25.184 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default", 2021-09-02T07:28:25.184 INFO:teuthology.orchestra.run.smithi175.stdout: "domain_root": "default.rgw.meta:root", 2021-09-02T07:28:25.185 INFO:teuthology.orchestra.run.smithi175.stdout: "control_pool": "default.rgw.control", 2021-09-02T07:28:25.185 INFO:teuthology.orchestra.run.smithi175.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-02T07:28:25.185 INFO:teuthology.orchestra.run.smithi175.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-02T07:28:25.185 INFO:teuthology.orchestra.run.smithi175.stdout: "log_pool": "default.rgw.log", 2021-09-02T07:28:25.186 INFO:teuthology.orchestra.run.smithi175.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-02T07:28:25.186 INFO:teuthology.orchestra.run.smithi175.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-02T07:28:25.186 INFO:teuthology.orchestra.run.smithi175.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-02T07:28:25.186 INFO:teuthology.orchestra.run.smithi175.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-02T07:28:25.186 INFO:teuthology.orchestra.run.smithi175.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-02T07:28:25.187 INFO:teuthology.orchestra.run.smithi175.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-02T07:28:25.187 INFO:teuthology.orchestra.run.smithi175.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-02T07:28:25.187 INFO:teuthology.orchestra.run.smithi175.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-02T07:28:25.187 INFO:teuthology.orchestra.run.smithi175.stdout: "otp_pool": "default.rgw.otp", 2021-09-02T07:28:25.188 INFO:teuthology.orchestra.run.smithi175.stdout: "system_key": { 2021-09-02T07:28:25.188 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "", 2021-09-02T07:28:25.188 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "" 2021-09-02T07:28:25.188 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:28:25.188 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_pools": [ 2021-09-02T07:28:25.189 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:28:25.189 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "default-placement", 2021-09-02T07:28:25.189 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:28:25.189 INFO:teuthology.orchestra.run.smithi175.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-02T07:28:25.190 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": { 2021-09-02T07:28:25.190 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD": { 2021-09-02T07:28:25.190 INFO:teuthology.orchestra.run.smithi175.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-02T07:28:25.190 INFO:teuthology.orchestra.run.smithi175.stdout: "compression_type": "random" 2021-09-02T07:28:25.190 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:28:25.191 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:28:25.191 INFO:teuthology.orchestra.run.smithi175.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-02T07:28:25.191 INFO:teuthology.orchestra.run.smithi175.stdout: "index_type": 0 2021-09-02T07:28:25.191 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:28:25.191 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:28:25.192 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:28:25.192 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "", 2021-09-02T07:28:25.192 INFO:teuthology.orchestra.run.smithi175.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-02T07:28:25.192 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:28:25.193 DEBUG:tasks.util.rgw: json result: {'id': '71986023-b31f-431b-9e99-0a21d50112a1', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-02T07:28:25.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2021-09-02T07:28:25.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2021-09-02T07:28:25.195 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2021-09-02T07:28:25.322 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:28:25.322 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:28:25.328 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:25.331+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.406 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.408+0000 7f8ed29e8380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:26.406 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.408+0000 7f8ed29e8380 20 realm 2021-09-02T07:28:26.407 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.408+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.409 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.408+0000 7f8ed29e8380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:26.409 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.408+0000 7f8ed29e8380 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-02T07:28:26.409 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.408+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.409 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.409+0000 7f8ed29e8380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:26.410 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.409+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.410 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.409+0000 7f8ed29e8380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:26.410 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.409+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.411 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.410+0000 7f8ed29e8380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:26.411 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.410+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.411 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.410+0000 7f8ed29e8380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:26.411 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.410+0000 7f8ed29e8380 10 Creating default zonegroup 2021-09-02T07:28:26.411 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.410+0000 7f8ed29e8380 20 get_system_obj_state: rctx=0x7ffc228f0ef0 obj=:.pools.avail state=0x563719e73d70 s->prefetch_data=0 2021-09-02T07:28:26.412 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.411+0000 7f8ed29e8380 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-09-02T07:28:26.422 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.425+0000 7f8ed29e8380 10 WARNING: store->list_zones() returned r=-2 2021-09-02T07:28:26.423 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.425+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.423 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.425+0000 7f8ed29e8380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:26.425 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.428+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.426 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.428+0000 7f8ed29e8380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:26.426 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.428+0000 7f8ed29e8380 10 could not read realm id: (2) No such file or directory 2021-09-02T07:28:26.427 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.428+0000 7f8ed29e8380 10 WARNING: failed to set zone as default, r=-22 2021-09-02T07:28:26.427 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.428+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.428 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.428+0000 7f8ed29e8380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:26.429 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.432+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.430 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.432+0000 7f8ed29e8380 20 rados_obj.operate() r=0 bl.length=873 2021-09-02T07:28:26.430 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.432+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.434 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.434+0000 7f8ed29e8380 20 rados_obj.operate() r=0 bl.length=358 2021-09-02T07:28:26.434 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.434+0000 7f8ed29e8380 10 Using default name default 2021-09-02T07:28:26.434 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.434+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.434 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.435+0000 7f8ed29e8380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:26.435 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.435+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.435 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.435+0000 7f8ed29e8380 20 rados_obj.operate() r=0 bl.length=873 2021-09-02T07:28:26.435 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.435+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.435 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.435+0000 7f8ed29e8380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:26.436 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.435+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.436 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.436+0000 7f8ed29e8380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:26.436 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.436+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.436 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.436+0000 7f8ed29e8380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:26.445 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.446+0000 7f8ed29e8380 20 RGWRados::pool_iterate: got zone_info.2cc3355f-0f21-4a02-9eb6-83efe5d59940 2021-09-02T07:28:26.445 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.446+0000 7f8ed29e8380 20 RGWRados::pool_iterate: got zonegroup_info.281d713b-43e2-47a5-b17f-16bf10cde7dd 2021-09-02T07:28:26.445 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.446+0000 7f8ed29e8380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-02T07:28:26.445 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.446+0000 7f8ed29e8380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-02T07:28:26.446 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.446+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.446 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.447+0000 7f8ed29e8380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:26.446 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.447+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.447 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.447+0000 7f8ed29e8380 20 rados_obj.operate() r=0 bl.length=873 2021-09-02T07:28:26.447 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.447+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.448 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.447+0000 7f8ed29e8380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:26.448 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.447+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.449 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.448+0000 7f8ed29e8380 20 rados_obj.operate() r=0 bl.length=358 2021-09-02T07:28:26.449 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.448+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.449 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.448+0000 7f8ed29e8380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:26.450 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.448+0000 7f8ed29e8380 20 zone default found 2021-09-02T07:28:26.450 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.448+0000 7f8ed29e8380 1 Realm: () 2021-09-02T07:28:26.451 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.448+0000 7f8ed29e8380 1 ZoneGroup: default (281d713b-43e2-47a5-b17f-16bf10cde7dd) 2021-09-02T07:28:26.451 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.448+0000 7f8ed29e8380 1 Zone: default (2cc3355f-0f21-4a02-9eb6-83efe5d59940) 2021-09-02T07:28:26.451 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.448+0000 7f8ed29e8380 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-02T07:28:26.452 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.448+0000 7f8ed29e8380 20 zonegroup default 2021-09-02T07:28:26.452 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.448+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:26.452 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.448+0000 7f8ed29e8380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:26.452 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:26.448+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:28.410 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:28.412+0000 7f8ed29e8380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:28.410 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:28.412+0000 7f8ed29e8380 20 rados->read ofs=0 len=0 2021-09-02T07:28:28.411 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:28.414+0000 7f8ed29e8380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:28.412 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:28.414+0000 7f8ed29e8380 20 started sync module instance, tier type = 2021-09-02T07:28:28.412 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:28.414+0000 7f8ed29e8380 20 started zone id=2cc3355f-0f21-4a02-9eb6-83efe5d59940 (name=default) with tier type = 2021-09-02T07:28:30.449 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.451+0000 7f8ed29e8380 20 add_watcher() i=0 2021-09-02T07:28:30.449 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.451+0000 7f8ed29e8380 20 add_watcher() i=1 2021-09-02T07:28:30.450 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.451+0000 7f8ed29e8380 20 add_watcher() i=2 2021-09-02T07:28:30.450 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.451+0000 7f8ed29e8380 20 add_watcher() i=3 2021-09-02T07:28:30.450 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.451+0000 7f8ed29e8380 20 add_watcher() i=4 2021-09-02T07:28:30.450 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.451+0000 7f8ed29e8380 20 add_watcher() i=5 2021-09-02T07:28:30.451 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.451+0000 7f8ed29e8380 20 add_watcher() i=6 2021-09-02T07:28:30.453 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.452+0000 7f8ed29e8380 20 add_watcher() i=7 2021-09-02T07:28:30.453 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.452+0000 7f8ed29e8380 2 all 8 watchers are set, enabling cache 2021-09-02T07:28:30.453 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.453+0000 7f8ed29e8380 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2021-09-02T07:28:30.453 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.453+0000 7f8ed29e8380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2021-09-02T07:28:30.454 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.453+0000 7f8ed29e8380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-02T07:28:30.454 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.453+0000 7f8ed29e8380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2021-09-02T07:28:30.454 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.453+0000 7f8ed29e8380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2021-09-02T07:28:30.454 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.453+0000 7f8ed29e8380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-02T07:28:30.455 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.453+0000 7f8ed29e8380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2021-09-02T07:28:30.455 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.453+0000 7f8ed29e8380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2021-09-02T07:28:30.455 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.453+0000 7f8ed29e8380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-02T07:28:30.455 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.454+0000 7f8ed29e8380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2021-09-02T07:28:30.456 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.454+0000 7f8ed29e8380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2021-09-02T07:28:30.456 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.454+0000 7f8ed29e8380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-02T07:28:30.456 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.454+0000 7f8ed29e8380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2021-09-02T07:28:30.456 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.454+0000 7f8ed29e8380 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2021-09-02T07:28:30.457 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.455+0000 7f8ed29e8380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-02T07:28:30.458 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.460+0000 7f8e96fe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-02T07:28:30.458 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.460+0000 7f8ed29e8380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-02T07:28:30.458 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:30.460+0000 7f8ed29e8380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-02T07:28:34.412 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.413+0000 7f8ed29e8380 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-09-02T07:28:34.412 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.413+0000 7f8ed29e8380 5 note: GC not initialized 2021-09-02T07:28:34.415 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.415+0000 7f8e15ffb700 20 reqs_thread_entry: start 2021-09-02T07:28:34.416 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.415+0000 7f8def7fe700 20 reqs_thread_entry: start 2021-09-02T07:28:34.416 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.415+0000 7f8ed29e8380 20 init_complete bucket index max shards: 11 2021-09-02T07:28:34.417 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.416+0000 7f8ed29e8380 10 rgw notify: Started notification manager with: 1 workers 2021-09-02T07:28:34.417 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.416+0000 7f8dedffb700 20 rgw notify: INFO: next queues processing will happen at: Thu Sep 2 07:29:04 2021 2021-09-02T07:28:34.418 INFO:teuthology.orchestra.run.smithi197.stderr: 2021-09-02T07:28:34.419 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.416+0000 7f8ded7fa700 20 reqs_thread_entry: start 2021-09-02T07:28:34.442 INFO:teuthology.orchestra.run.smithi197.stdout:[] 2021-09-02T07:28:34.444 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.443+0000 7f8ed29e8380 20 remove_watcher() i=0 2021-09-02T07:28:34.444 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.443+0000 7f8ed29e8380 2 removed watcher, disabling cache 2021-09-02T07:28:34.444 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.444+0000 7f8ed29e8380 20 remove_watcher() i=1 2021-09-02T07:28:34.445 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.445+0000 7f8ed29e8380 20 remove_watcher() i=2 2021-09-02T07:28:34.445 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.447+0000 7f8ed29e8380 20 remove_watcher() i=3 2021-09-02T07:28:34.446 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.448+0000 7f8ed29e8380 20 remove_watcher() i=4 2021-09-02T07:28:34.448 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.449+0000 7f8ed29e8380 20 remove_watcher() i=5 2021-09-02T07:28:34.450 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.451+0000 7f8ed29e8380 20 remove_watcher() i=6 2021-09-02T07:28:34.451 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.453+0000 7f8ed29e8380 20 remove_watcher() i=7 2021-09-02T07:28:34.453 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.453+0000 7f8e96fe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-02T07:28:34.458 DEBUG:tasks.util.rgw: json result: [] 2021-09-02T07:28:34.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-02T07:28:34.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-02T07:28:34.459 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-02T07:28:34.506 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:28:34.507 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:28:34.513 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.513+0000 7f52519a1380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-02T07:28:34.514 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.513+0000 7f52519a1380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-02T07:28:34.515 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.517+0000 7f518e7fc700 20 reqs_thread_entry: start 2021-09-02T07:28:34.515 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.517+0000 7f52519a1380 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-02T07:28:34.516 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.517+0000 7f52519a1380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.516 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.518+0000 7f52519a1380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:34.517 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.518+0000 7f52519a1380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.517 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.518+0000 7f52519a1380 20 rados_obj.operate() r=0 bl.length=873 2021-09-02T07:28:34.528 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.529+0000 7f52519a1380 20 RGWRados::pool_iterate: got zone_info.2cc3355f-0f21-4a02-9eb6-83efe5d59940 2021-09-02T07:28:34.528 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.529+0000 7f52519a1380 20 RGWRados::pool_iterate: got zonegroup_info.281d713b-43e2-47a5-b17f-16bf10cde7dd 2021-09-02T07:28:34.528 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.529+0000 7f52519a1380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-02T07:28:34.529 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.529+0000 7f52519a1380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-02T07:28:34.529 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.529+0000 7f52519a1380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.529 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.529+0000 7f52519a1380 20 rados_obj.operate() r=0 bl.length=873 2021-09-02T07:28:34.530 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.529+0000 7f52519a1380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.530 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.529+0000 7f52519a1380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:34.530 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.529+0000 7f52519a1380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.531 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.530+0000 7f52519a1380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:34.531 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.530+0000 7f52519a1380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.531 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.530+0000 7f52519a1380 20 rados_obj.operate() r=0 bl.length=358 2021-09-02T07:28:34.533 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:28:34.533 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "2cc3355f-0f21-4a02-9eb6-83efe5d59940", 2021-09-02T07:28:34.533 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "default", 2021-09-02T07:28:34.533 INFO:teuthology.orchestra.run.smithi197.stdout: "domain_root": "default.rgw.meta:root", 2021-09-02T07:28:34.534 INFO:teuthology.orchestra.run.smithi197.stdout: "control_pool": "default.rgw.control", 2021-09-02T07:28:34.534 INFO:teuthology.orchestra.run.smithi197.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-02T07:28:34.534 INFO:teuthology.orchestra.run.smithi197.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-02T07:28:34.534 INFO:teuthology.orchestra.run.smithi197.stdout: "log_pool": "default.rgw.log", 2021-09-02T07:28:34.534 INFO:teuthology.orchestra.run.smithi197.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-02T07:28:34.535 INFO:teuthology.orchestra.run.smithi197.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-02T07:28:34.535 INFO:teuthology.orchestra.run.smithi197.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-02T07:28:34.535 INFO:teuthology.orchestra.run.smithi197.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-02T07:28:34.535 INFO:teuthology.orchestra.run.smithi197.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-02T07:28:34.536 INFO:teuthology.orchestra.run.smithi197.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-02T07:28:34.536 INFO:teuthology.orchestra.run.smithi197.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-02T07:28:34.536 INFO:teuthology.orchestra.run.smithi197.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-02T07:28:34.536 INFO:teuthology.orchestra.run.smithi197.stdout: "otp_pool": "default.rgw.otp", 2021-09-02T07:28:34.536 INFO:teuthology.orchestra.run.smithi197.stdout: "system_key": { 2021-09-02T07:28:34.537 INFO:teuthology.orchestra.run.smithi197.stdout: "access_key": "", 2021-09-02T07:28:34.537 INFO:teuthology.orchestra.run.smithi197.stdout: "secret_key": "" 2021-09-02T07:28:34.537 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:28:34.537 INFO:teuthology.orchestra.run.smithi197.stdout: "placement_pools": [ 2021-09-02T07:28:34.538 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:28:34.538 INFO:teuthology.orchestra.run.smithi197.stdout: "key": "default-placement", 2021-09-02T07:28:34.538 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:28:34.538 INFO:teuthology.orchestra.run.smithi197.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-02T07:28:34.539 INFO:teuthology.orchestra.run.smithi197.stdout: "storage_classes": { 2021-09-02T07:28:34.539 INFO:teuthology.orchestra.run.smithi197.stdout: "STANDARD": { 2021-09-02T07:28:34.539 INFO:teuthology.orchestra.run.smithi197.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-02T07:28:34.540 INFO:teuthology.orchestra.run.smithi197.stdout: "compression_type": "random" 2021-09-02T07:28:34.540 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:28:34.540 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:28:34.540 INFO:teuthology.orchestra.run.smithi197.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-02T07:28:34.541 INFO:teuthology.orchestra.run.smithi197.stdout: "index_type": 0 2021-09-02T07:28:34.541 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:28:34.541 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:28:34.541 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:28:34.541 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_id": "", 2021-09-02T07:28:34.542 INFO:teuthology.orchestra.run.smithi197.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-02T07:28:34.542 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:28:34.543 DEBUG:tasks.util.rgw: json result: {'id': '2cc3355f-0f21-4a02-9eb6-83efe5d59940', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-02T07:28:34.544 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2021-09-02T07:28:34.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2021-09-02T07:28:34.544 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2021-09-02T07:28:34.581 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:28:34.582 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:28:34.586 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.588+0000 7f927fc8a380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.588 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.589+0000 7f927fc8a380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:34.588 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.589+0000 7f927fc8a380 20 realm 2021-09-02T07:28:34.589 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.589+0000 7f927fc8a380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.589 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.590+0000 7f927fc8a380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:34.590 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.590+0000 7f927fc8a380 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-02T07:28:34.590 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.590+0000 7f927fc8a380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.590 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.590+0000 7f927fc8a380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:34.590 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.590+0000 7f927fc8a380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.591 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.590+0000 7f927fc8a380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:34.592 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.590+0000 7f927fc8a380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.592 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.591+0000 7f927fc8a380 20 rados_obj.operate() r=0 bl.length=889 2021-09-02T07:28:34.592 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.591+0000 7f927fc8a380 20 searching for the correct realm 2021-09-02T07:28:34.601 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.601+0000 7f927fc8a380 20 RGWRados::pool_iterate: got zone_info.2cc3355f-0f21-4a02-9eb6-83efe5d59940 2021-09-02T07:28:34.601 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.601+0000 7f927fc8a380 20 RGWRados::pool_iterate: got zonegroup_info.281d713b-43e2-47a5-b17f-16bf10cde7dd 2021-09-02T07:28:34.602 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.601+0000 7f927fc8a380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-02T07:28:34.602 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.601+0000 7f927fc8a380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-02T07:28:34.602 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.601+0000 7f927fc8a380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.602 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.601+0000 7f927fc8a380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:34.602 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.601+0000 7f927fc8a380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.603 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.601+0000 7f927fc8a380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:34.603 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.601+0000 7f927fc8a380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.603 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.602+0000 7f927fc8a380 20 rados_obj.operate() r=0 bl.length=358 2021-09-02T07:28:34.603 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.602+0000 7f927fc8a380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.604 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.602+0000 7f927fc8a380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:34.604 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.602+0000 7f927fc8a380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.604 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.602+0000 7f927fc8a380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:34.604 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.602+0000 7f927fc8a380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.604 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.603+0000 7f927fc8a380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:34.614 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.611+0000 7f927fc8a380 20 RGWRados::pool_iterate: got zone_info.2cc3355f-0f21-4a02-9eb6-83efe5d59940 2021-09-02T07:28:34.615 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.611+0000 7f927fc8a380 20 RGWRados::pool_iterate: got zonegroup_info.281d713b-43e2-47a5-b17f-16bf10cde7dd 2021-09-02T07:28:34.615 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.611+0000 7f927fc8a380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-02T07:28:34.615 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.611+0000 7f927fc8a380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-02T07:28:34.615 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.611+0000 7f927fc8a380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.616 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.611+0000 7f927fc8a380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:34.616 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.611+0000 7f927fc8a380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.616 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.612+0000 7f927fc8a380 20 rados_obj.operate() r=0 bl.length=889 2021-09-02T07:28:34.616 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.612+0000 7f927fc8a380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.617 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.612+0000 7f927fc8a380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:34.617 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.612+0000 7f927fc8a380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.617 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.613+0000 7f927fc8a380 20 rados_obj.operate() r=0 bl.length=358 2021-09-02T07:28:34.617 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.613+0000 7f927fc8a380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.618 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.613+0000 7f927fc8a380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:34.618 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.613+0000 7f927fc8a380 20 zone default found 2021-09-02T07:28:34.618 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.613+0000 7f927fc8a380 1 Realm: () 2021-09-02T07:28:34.618 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.613+0000 7f927fc8a380 1 ZoneGroup: default (281d713b-43e2-47a5-b17f-16bf10cde7dd) 2021-09-02T07:28:34.618 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.613+0000 7f927fc8a380 1 Zone: default (2cc3355f-0f21-4a02-9eb6-83efe5d59940) 2021-09-02T07:28:34.619 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.613+0000 7f927fc8a380 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-02T07:28:34.619 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.613+0000 7f927fc8a380 20 zonegroup default 2021-09-02T07:28:34.619 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.613+0000 7f927fc8a380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.619 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.613+0000 7f927fc8a380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:34.620 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.613+0000 7f927fc8a380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.620 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.614+0000 7f927fc8a380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:34.620 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.614+0000 7f927fc8a380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.620 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.614+0000 7f927fc8a380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:34.620 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.614+0000 7f927fc8a380 20 started sync module instance, tier type = 2021-09-02T07:28:34.621 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.614+0000 7f927fc8a380 20 started zone id=2cc3355f-0f21-4a02-9eb6-83efe5d59940 (name=default) with tier type = 2021-09-02T07:28:34.632 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.633+0000 7f927fc8a380 20 add_watcher() i=0 2021-09-02T07:28:34.632 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.633+0000 7f927fc8a380 20 add_watcher() i=1 2021-09-02T07:28:34.633 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.633+0000 7f927fc8a380 20 add_watcher() i=2 2021-09-02T07:28:34.633 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.633+0000 7f927fc8a380 20 add_watcher() i=3 2021-09-02T07:28:34.633 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.633+0000 7f927fc8a380 20 add_watcher() i=4 2021-09-02T07:28:34.633 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.633+0000 7f927fc8a380 20 add_watcher() i=5 2021-09-02T07:28:34.633 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.633+0000 7f927fc8a380 20 add_watcher() i=6 2021-09-02T07:28:34.634 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.634+0000 7f927fc8a380 20 add_watcher() i=7 2021-09-02T07:28:34.634 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.634+0000 7f927fc8a380 2 all 8 watchers are set, enabling cache 2021-09-02T07:28:34.635 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.636+0000 7f91bffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-02T07:28:34.635 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.636+0000 7f927fc8a380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-02T07:28:34.636 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.636+0000 7f927fc8a380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-02T07:28:34.638 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.640+0000 7f927fc8a380 5 note: GC not initialized 2021-09-02T07:28:34.640 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.641+0000 7f91aeffd700 20 reqs_thread_entry: start 2021-09-02T07:28:34.640 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.641+0000 7f91ad7fa700 20 reqs_thread_entry: start 2021-09-02T07:28:34.640 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.641+0000 7f927fc8a380 20 init_complete bucket index max shards: 11 2021-09-02T07:28:34.640 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.641+0000 7f927fc8a380 10 rgw notify: Started notification manager with: 1 workers 2021-09-02T07:28:34.641 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.641+0000 7f91a6ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Sep 2 07:29:04 2021 2021-09-02T07:28:34.641 INFO:teuthology.orchestra.run.smithi197.stderr: 2021-09-02T07:28:34.641 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.641+0000 7f91a67fc700 20 reqs_thread_entry: start 2021-09-02T07:28:34.651 INFO:teuthology.orchestra.run.smithi197.stdout:[] 2021-09-02T07:28:34.651 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.652+0000 7f927fc8a380 20 remove_watcher() i=0 2021-09-02T07:28:34.651 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.652+0000 7f927fc8a380 2 removed watcher, disabling cache 2021-09-02T07:28:34.652 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.653+0000 7f927fc8a380 20 remove_watcher() i=1 2021-09-02T07:28:34.653 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.654+0000 7f927fc8a380 20 remove_watcher() i=2 2021-09-02T07:28:34.654 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.655+0000 7f927fc8a380 20 remove_watcher() i=3 2021-09-02T07:28:34.655 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.656+0000 7f927fc8a380 20 remove_watcher() i=4 2021-09-02T07:28:34.656 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.658+0000 7f927fc8a380 20 remove_watcher() i=5 2021-09-02T07:28:34.657 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.659+0000 7f927fc8a380 20 remove_watcher() i=6 2021-09-02T07:28:34.658 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.660+0000 7f927fc8a380 20 remove_watcher() i=7 2021-09-02T07:28:34.659 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.661+0000 7f91bffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-02T07:28:34.664 DEBUG:tasks.util.rgw: json result: [] 2021-09-02T07:28:34.665 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-02T07:28:34.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-02T07:28:34.666 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-02T07:28:34.746 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:28:34.747 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:28:34.751 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.753+0000 7f6b521fc380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-02T07:28:34.752 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.753+0000 7f6b521fc380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-02T07:28:34.755 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.756+0000 7f6a8e7fc700 20 reqs_thread_entry: start 2021-09-02T07:28:34.755 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.756+0000 7f6b521fc380 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-02T07:28:34.755 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.756+0000 7f6b521fc380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.756 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.757+0000 7f6b521fc380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:34.756 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.757+0000 7f6b521fc380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.757 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.758+0000 7f6b521fc380 20 rados_obj.operate() r=0 bl.length=889 2021-09-02T07:28:34.768 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.769+0000 7f6b521fc380 20 RGWRados::pool_iterate: got zone_info.2cc3355f-0f21-4a02-9eb6-83efe5d59940 2021-09-02T07:28:34.768 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.769+0000 7f6b521fc380 20 RGWRados::pool_iterate: got zonegroup_info.281d713b-43e2-47a5-b17f-16bf10cde7dd 2021-09-02T07:28:34.768 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.769+0000 7f6b521fc380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-02T07:28:34.768 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.769+0000 7f6b521fc380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-02T07:28:34.769 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.769+0000 7f6b521fc380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.770 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.769+0000 7f6b521fc380 20 rados_obj.operate() r=0 bl.length=889 2021-09-02T07:28:34.770 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.769+0000 7f6b521fc380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.770 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.770+0000 7f6b521fc380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-02T07:28:34.771 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.770+0000 7f6b521fc380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.771 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.770+0000 7f6b521fc380 20 rados_obj.operate() r=0 bl.length=46 2021-09-02T07:28:34.771 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.770+0000 7f6b521fc380 20 rados->read ofs=0 len=0 2021-09-02T07:28:34.771 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:34.770+0000 7f6b521fc380 20 rados_obj.operate() r=0 bl.length=358 2021-09-02T07:28:34.773 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:28:34.773 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "2cc3355f-0f21-4a02-9eb6-83efe5d59940", 2021-09-02T07:28:34.774 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "default", 2021-09-02T07:28:34.774 INFO:teuthology.orchestra.run.smithi197.stdout: "domain_root": "default.rgw.meta:root", 2021-09-02T07:28:34.774 INFO:teuthology.orchestra.run.smithi197.stdout: "control_pool": "default.rgw.control", 2021-09-02T07:28:34.774 INFO:teuthology.orchestra.run.smithi197.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-02T07:28:34.775 INFO:teuthology.orchestra.run.smithi197.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-02T07:28:34.775 INFO:teuthology.orchestra.run.smithi197.stdout: "log_pool": "default.rgw.log", 2021-09-02T07:28:34.775 INFO:teuthology.orchestra.run.smithi197.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-02T07:28:34.775 INFO:teuthology.orchestra.run.smithi197.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-02T07:28:34.776 INFO:teuthology.orchestra.run.smithi197.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-02T07:28:34.776 INFO:teuthology.orchestra.run.smithi197.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-02T07:28:34.776 INFO:teuthology.orchestra.run.smithi197.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-02T07:28:34.776 INFO:teuthology.orchestra.run.smithi197.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-02T07:28:34.776 INFO:teuthology.orchestra.run.smithi197.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-02T07:28:34.777 INFO:teuthology.orchestra.run.smithi197.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-02T07:28:34.777 INFO:teuthology.orchestra.run.smithi197.stdout: "otp_pool": "default.rgw.otp", 2021-09-02T07:28:34.777 INFO:teuthology.orchestra.run.smithi197.stdout: "system_key": { 2021-09-02T07:28:34.777 INFO:teuthology.orchestra.run.smithi197.stdout: "access_key": "", 2021-09-02T07:28:34.778 INFO:teuthology.orchestra.run.smithi197.stdout: "secret_key": "" 2021-09-02T07:28:34.779 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:28:34.779 INFO:teuthology.orchestra.run.smithi197.stdout: "placement_pools": [ 2021-09-02T07:28:34.779 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:28:34.779 INFO:teuthology.orchestra.run.smithi197.stdout: "key": "default-placement", 2021-09-02T07:28:34.780 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:28:34.780 INFO:teuthology.orchestra.run.smithi197.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-02T07:28:34.780 INFO:teuthology.orchestra.run.smithi197.stdout: "storage_classes": { 2021-09-02T07:28:34.780 INFO:teuthology.orchestra.run.smithi197.stdout: "STANDARD": { 2021-09-02T07:28:34.781 INFO:teuthology.orchestra.run.smithi197.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-02T07:28:34.781 INFO:teuthology.orchestra.run.smithi197.stdout: "compression_type": "random" 2021-09-02T07:28:34.781 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:28:34.782 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:28:34.782 INFO:teuthology.orchestra.run.smithi197.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-02T07:28:34.782 INFO:teuthology.orchestra.run.smithi197.stdout: "index_type": 0 2021-09-02T07:28:34.782 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:28:34.782 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:28:34.783 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:28:34.783 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_id": "", 2021-09-02T07:28:34.783 INFO:teuthology.orchestra.run.smithi197.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-02T07:28:34.783 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:28:34.784 DEBUG:tasks.util.rgw: json result: {'id': '2cc3355f-0f21-4a02-9eb6-83efe5d59940', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-02T07:28:34.785 INFO:tasks.rgw:Starting rgw... 2021-09-02T07:28:34.785 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2021-09-02T07:28:34.786 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-02T07:28:34.786 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-09-02T07:28:34.787 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-09-02T07:28:34.790 INFO:tasks.rgw.c1.client.0:Started 2021-09-02T07:28:34.790 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2021-09-02T07:28:34.790 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-02T07:28:34.791 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-09-02T07:28:34.792 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-09-02T07:28:34.794 INFO:tasks.rgw.c1.client.1:Started 2021-09-02T07:28:34.795 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2021-09-02T07:28:34.795 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-02T07:28:34.795 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-09-02T07:28:34.796 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-09-02T07:28:34.827 INFO:tasks.rgw.c2.client.0:Started 2021-09-02T07:28:34.827 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2021-09-02T07:28:34.828 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-02T07:28:34.828 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-09-02T07:28:34.828 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-09-02T07:28:34.830 INFO:tasks.rgw.c2.client.1:Started 2021-09-02T07:28:34.831 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi175.front.sepia.ceph.com:8000/ 2021-09-02T07:28:34.831 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8000/ 2021-09-02T07:28:34.868 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-02T07:28:34.870 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-02T07:28:34.871 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2021-09-02T07:28:34.871 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi175.front.sepia.ceph.com port 8000: Connection refused 2021-09-02T07:28:35.872 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8000/ 2021-09-02T07:28:35.890 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-02T07:28:35.890 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2021-09-02T07:28:35.892 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2021-09-02T07:28:35.893 INFO:teuthology.orchestra.run.smithi175.stdout:anonymous 2021-09-02T07:28:35.893 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi175.front.sepia.ceph.com:8001/ 2021-09-02T07:28:35.894 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8001/ 2021-09-02T07:28:35.951 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-02T07:28:35.952 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2021-09-02T07:28:35.952 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2021-09-02T07:28:35.953 INFO:teuthology.orchestra.run.smithi175.stdout:anonymous 2021-09-02T07:28:35.953 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi197.front.sepia.ceph.com:8000/ 2021-09-02T07:28:35.954 DEBUG:teuthology.orchestra.run.smithi197:> curl http://smithi197.front.sepia.ceph.com:8000/ 2021-09-02T07:28:35.995 INFO:teuthology.orchestra.run.smithi197.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-02T07:28:35.996 INFO:teuthology.orchestra.run.smithi197.stderr: Dload Upload Total Spent Left Speed 2021-09-02T07:28:35.997 INFO:teuthology.orchestra.run.smithi197.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2021-09-02T07:28:35.998 INFO:teuthology.orchestra.run.smithi197.stdout:anonymous 2021-09-02T07:28:35.998 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi197.front.sepia.ceph.com:8001/ 2021-09-02T07:28:35.999 DEBUG:teuthology.orchestra.run.smithi197:> curl http://smithi197.front.sepia.ceph.com:8001/ 2021-09-02T07:28:36.059 INFO:teuthology.orchestra.run.smithi197.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-02T07:28:36.059 INFO:teuthology.orchestra.run.smithi197.stderr: Dload Upload Total Spent Left Speed 2021-09-02T07:28:36.059 INFO:teuthology.orchestra.run.smithi197.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2021-09-02T07:28:36.060 INFO:teuthology.orchestra.run.smithi197.stdout:anonymous 2021-09-02T07:28:36.061 INFO:teuthology.run_tasks:Running task rgw-multisite... 2021-09-02T07:28:36.327 INFO:tasks.rgw_multisite:creating realm.. 2021-09-02T07:28:36.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:28:36.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:28:36.328 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-02T07:28:36.368 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:28:36.369 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:28:36.402 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:28:36.402 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T07:28:36.402 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-realm", 2021-09-02T07:28:36.403 INFO:teuthology.orchestra.run.smithi175.stdout: "current_period": "dcf62b62-291a-46a6-867e-5263b7775f6a", 2021-09-02T07:28:36.403 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 1 2021-09-02T07:28:36.403 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:28:36.404 INFO:tasks.rgw_multisite:creating master zonegroup.. 2021-09-02T07:28:36.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:28:36.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:28:36.406 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-02T07:28:36.485 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:28:36.485 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:28:36.496 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:36.498+0000 7f98285f9380 -1 old format 2021-09-02T07:28:36.505 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:28:36.506 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2021-09-02T07:28:36.506 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2021-09-02T07:28:36.506 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2021-09-02T07:28:36.507 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": "true", 2021-09-02T07:28:36.507 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2021-09-02T07:28:36.507 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2021-09-02T07:28:36.507 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:28:36.508 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2021-09-02T07:28:36.508 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2021-09-02T07:28:36.508 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "", 2021-09-02T07:28:36.508 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [], 2021-09-02T07:28:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [], 2021-09-02T07:28:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "", 2021-09-02T07:28:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T07:28:36.509 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2021-09-02T07:28:36.510 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2021-09-02T07:28:36.510 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:28:36.510 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:28:36.510 INFO:tasks.rgw_multisite:creating master zone.. 2021-09-02T07:28:36.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'YDUXR4Y826KQKR5A', '--secret', 'DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w', '--endpoints', 'http://smithi175.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:28:36.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'YDUXR4Y826KQKR5A', '--secret', 'DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w', '--endpoints', 'http://smithi175.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:28:36.512 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key YDUXR4Y826KQKR5A --secret DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w --endpoints http://smithi175.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-02T07:28:36.593 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:28:36.594 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:28:36.628 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:28:36.628 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:28:36.628 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2021-09-02T07:28:36.628 INFO:teuthology.orchestra.run.smithi175.stdout: "domain_root": "test-zone1.rgw.meta:root", 2021-09-02T07:28:36.629 INFO:teuthology.orchestra.run.smithi175.stdout: "control_pool": "test-zone1.rgw.control", 2021-09-02T07:28:36.629 INFO:teuthology.orchestra.run.smithi175.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2021-09-02T07:28:36.629 INFO:teuthology.orchestra.run.smithi175.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2021-09-02T07:28:36.629 INFO:teuthology.orchestra.run.smithi175.stdout: "log_pool": "test-zone1.rgw.log", 2021-09-02T07:28:36.630 INFO:teuthology.orchestra.run.smithi175.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2021-09-02T07:28:36.630 INFO:teuthology.orchestra.run.smithi175.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2021-09-02T07:28:36.630 INFO:teuthology.orchestra.run.smithi175.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2021-09-02T07:28:36.630 INFO:teuthology.orchestra.run.smithi175.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2021-09-02T07:28:36.631 INFO:teuthology.orchestra.run.smithi175.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2021-09-02T07:28:36.631 INFO:teuthology.orchestra.run.smithi175.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2021-09-02T07:28:36.631 INFO:teuthology.orchestra.run.smithi175.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2021-09-02T07:28:36.631 INFO:teuthology.orchestra.run.smithi175.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2021-09-02T07:28:36.632 INFO:teuthology.orchestra.run.smithi175.stdout: "otp_pool": "test-zone1.rgw.otp", 2021-09-02T07:28:36.632 INFO:teuthology.orchestra.run.smithi175.stdout: "system_key": { 2021-09-02T07:28:36.632 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "YDUXR4Y826KQKR5A", 2021-09-02T07:28:36.633 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w" 2021-09-02T07:28:36.633 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:28:36.633 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_pools": [ 2021-09-02T07:28:36.633 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:28:36.633 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "default-placement", 2021-09-02T07:28:36.634 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:28:36.634 INFO:teuthology.orchestra.run.smithi175.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2021-09-02T07:28:36.634 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": { 2021-09-02T07:28:36.634 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD": { 2021-09-02T07:28:36.635 INFO:teuthology.orchestra.run.smithi175.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2021-09-02T07:28:36.635 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:28:36.635 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:28:36.635 INFO:teuthology.orchestra.run.smithi175.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2021-09-02T07:28:36.635 INFO:teuthology.orchestra.run.smithi175.stdout: "index_type": 0 2021-09-02T07:28:36.636 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:28:36.636 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:28:36.638 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:28:36.638 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T07:28:36.638 INFO:teuthology.orchestra.run.smithi175.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2021-09-02T07:28:36.638 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:28:36.640 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2021-09-02T07:28:37.821 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'test-zone1.rgw.buckets.data' created 2021-09-02T07:28:37.838 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2021-09-02T07:28:38.844 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2021-09-02T07:28:38.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:28:38.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:28:38.860 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-02T07:28:38.915 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:28:38.916 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:28:38.944 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:28:38.944 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:28:38.945 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2021-09-02T07:28:38.945 INFO:teuthology.orchestra.run.smithi175.stdout: "domain_root": "test-zone1.rgw.meta:root", 2021-09-02T07:28:38.945 INFO:teuthology.orchestra.run.smithi175.stdout: "control_pool": "test-zone1.rgw.control", 2021-09-02T07:28:38.945 INFO:teuthology.orchestra.run.smithi175.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2021-09-02T07:28:38.945 INFO:teuthology.orchestra.run.smithi175.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2021-09-02T07:28:38.946 INFO:teuthology.orchestra.run.smithi175.stdout: "log_pool": "test-zone1.rgw.log", 2021-09-02T07:28:38.946 INFO:teuthology.orchestra.run.smithi175.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2021-09-02T07:28:38.946 INFO:teuthology.orchestra.run.smithi175.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2021-09-02T07:28:38.947 INFO:teuthology.orchestra.run.smithi175.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2021-09-02T07:28:38.947 INFO:teuthology.orchestra.run.smithi175.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2021-09-02T07:28:38.947 INFO:teuthology.orchestra.run.smithi175.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2021-09-02T07:28:38.947 INFO:teuthology.orchestra.run.smithi175.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2021-09-02T07:28:38.947 INFO:teuthology.orchestra.run.smithi175.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2021-09-02T07:28:38.948 INFO:teuthology.orchestra.run.smithi175.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2021-09-02T07:28:38.948 INFO:teuthology.orchestra.run.smithi175.stdout: "otp_pool": "test-zone1.rgw.otp", 2021-09-02T07:28:38.948 INFO:teuthology.orchestra.run.smithi175.stdout: "system_key": { 2021-09-02T07:28:38.948 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "YDUXR4Y826KQKR5A", 2021-09-02T07:28:38.949 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w" 2021-09-02T07:28:38.949 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:28:38.949 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_pools": [ 2021-09-02T07:28:38.949 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:28:38.950 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "default-placement", 2021-09-02T07:28:38.950 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:28:38.950 INFO:teuthology.orchestra.run.smithi175.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2021-09-02T07:28:38.951 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": { 2021-09-02T07:28:38.951 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD": { 2021-09-02T07:28:38.951 INFO:teuthology.orchestra.run.smithi175.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2021-09-02T07:28:38.951 INFO:teuthology.orchestra.run.smithi175.stdout: "compression_type": "random" 2021-09-02T07:28:38.951 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:28:38.952 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:28:38.952 INFO:teuthology.orchestra.run.smithi175.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2021-09-02T07:28:38.952 INFO:teuthology.orchestra.run.smithi175.stdout: "index_type": 0 2021-09-02T07:28:38.952 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:28:38.953 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:28:38.953 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:28:38.954 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "", 2021-09-02T07:28:38.954 INFO:teuthology.orchestra.run.smithi175.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2021-09-02T07:28:38.954 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:28:38.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:28:38.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:28:38.956 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-02T07:28:38.998 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:28:38.999 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:28:39.008 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:28:39.009+0000 7fe0504cb380 0 period (dcf62b62-291a-46a6-867e-5263b7775f6a does not have zone 8b6c886f-f9c4-4c5c-b256-edd3d1362f1c configured 2021-09-02T07:28:46.930 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:28:46.930 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:28:46.931 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 1, 2021-09-02T07:28:46.931 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "dcf62b62-291a-46a6-867e-5263b7775f6a", 2021-09-02T07:28:46.931 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2021-09-02T07:28:46.931 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2021-09-02T07:28:46.932 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:28:46.932 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2021-09-02T07:28:46.932 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:28:46.932 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2021-09-02T07:28:46.933 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2021-09-02T07:28:46.933 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2021-09-02T07:28:46.933 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": "true", 2021-09-02T07:28:46.933 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2021-09-02T07:28:46.933 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2021-09-02T07:28:46.934 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:28:46.934 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2021-09-02T07:28:46.934 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2021-09-02T07:28:46.934 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:28:46.935 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2021-09-02T07:28:46.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:28:46.935 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:28:46.935 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2021-09-02T07:28:46.936 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2021-09-02T07:28:46.936 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2021-09-02T07:28:46.936 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:28:46.936 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": "false", 2021-09-02T07:28:46.937 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": "false", 2021-09-02T07:28:46.937 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2021-09-02T07:28:46.937 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": "false", 2021-09-02T07:28:46.938 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2021-09-02T07:28:46.938 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": "true", 2021-09-02T07:28:46.938 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2021-09-02T07:28:46.938 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "" 2021-09-02T07:28:46.939 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:28:46.939 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:28:46.939 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2021-09-02T07:28:46.939 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:28:46.940 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2021-09-02T07:28:46.940 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2021-09-02T07:28:46.940 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2021-09-02T07:28:46.940 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2021-09-02T07:28:46.941 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:28:46.941 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:28:46.941 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:28:46.941 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2021-09-02T07:28:46.941 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T07:28:46.942 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2021-09-02T07:28:46.942 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2021-09-02T07:28:46.942 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:28:46.943 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:28:46.943 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:28:46.943 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2021-09-02T07:28:46.944 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:28:46.944 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:28:46.944 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2303553998 2021-09-02T07:28:46.944 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:28:46.945 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:28:46.945 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:28:46.945 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2021-09-02T07:28:46.945 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:28:46.945 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2021-09-02T07:28:46.946 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2021-09-02T07:28:46.946 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2021-09-02T07:28:46.947 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2021-09-02T07:28:46.947 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2021-09-02T07:28:46.947 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2021-09-02T07:28:46.947 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2021-09-02T07:28:46.948 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:28:46.948 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2021-09-02T07:28:46.948 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2021-09-02T07:28:46.948 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2021-09-02T07:28:46.948 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2021-09-02T07:28:46.949 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2021-09-02T07:28:46.949 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2021-09-02T07:28:46.949 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:28:46.950 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:28:46.950 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T07:28:46.950 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2021-09-02T07:28:46.950 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:28:46.951 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:28:46.952 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2021-09-02T07:28:46.953 INFO:teuthology.orchestra.run:waiting for 300 2021-09-02T07:28:46.993 INFO:tasks.rgw.c1.client.0.smithi175.stdout:2021-09-02T07:28:46.995+0000 7f6c1e331700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 35560) UID: 0 2021-09-02T07:28:46.994 INFO:tasks.rgw.c1.client.0.smithi175.stdout:2021-09-02T07:28:46.995+0000 7f6c2f26c440 -1 shutting down 2021-09-02T07:28:49.728 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-09-02T07:28:53.054 INFO:tasks.rgw.c1.client.0:Stopped 2021-09-02T07:28:53.055 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-09-02T07:28:53.055 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-09-02T07:28:53.059 INFO:tasks.rgw.c1.client.0:Started 2021-09-02T07:28:53.060 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8000 2021-09-02T07:28:53.083 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-02T07:28:53.083 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-02T07:28:53.084 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2021-09-02T07:28:53.084 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi175.front.sepia.ceph.com port 8000: Connection refused 2021-09-02T07:28:54.085 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8000 2021-09-02T07:28:54.105 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-02T07:28:54.106 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2021-09-02T07:28:54.106 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2021-09-02T07:28:54.107 INFO:teuthology.orchestra.run.smithi175.stdout:anonymous 2021-09-02T07:28:54.108 INFO:tasks.rgw_multisite:creating admin user.. 2021-09-02T07:28:54.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'YDUXR4Y826KQKR5A', '--secret', 'DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:28:54.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'YDUXR4Y826KQKR5A', '--secret', 'DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:28:54.109 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key YDUXR4Y826KQKR5A --secret DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-02T07:28:54.197 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:28:54.197 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:28:54.242 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:28:54.243 INFO:teuthology.orchestra.run.smithi175.stdout: "user_id": "realm-admin", 2021-09-02T07:28:54.243 INFO:teuthology.orchestra.run.smithi175.stdout: "display_name": "Realm Admin", 2021-09-02T07:28:54.243 INFO:teuthology.orchestra.run.smithi175.stdout: "email": "", 2021-09-02T07:28:54.243 INFO:teuthology.orchestra.run.smithi175.stdout: "suspended": 0, 2021-09-02T07:28:54.244 INFO:teuthology.orchestra.run.smithi175.stdout: "max_buckets": 1000, 2021-09-02T07:28:54.244 INFO:teuthology.orchestra.run.smithi175.stdout: "subusers": [], 2021-09-02T07:28:54.244 INFO:teuthology.orchestra.run.smithi175.stdout: "keys": [ 2021-09-02T07:28:54.244 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:28:54.245 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "realm-admin", 2021-09-02T07:28:54.245 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "YDUXR4Y826KQKR5A", 2021-09-02T07:28:54.245 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w" 2021-09-02T07:28:54.245 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:28:54.246 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:28:54.246 INFO:teuthology.orchestra.run.smithi175.stdout: "swift_keys": [], 2021-09-02T07:28:54.246 INFO:teuthology.orchestra.run.smithi175.stdout: "caps": [], 2021-09-02T07:28:54.246 INFO:teuthology.orchestra.run.smithi175.stdout: "op_mask": "read, write, delete", 2021-09-02T07:28:54.246 INFO:teuthology.orchestra.run.smithi175.stdout: "system": "true", 2021-09-02T07:28:54.247 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "", 2021-09-02T07:28:54.247 INFO:teuthology.orchestra.run.smithi175.stdout: "default_storage_class": "", 2021-09-02T07:28:54.247 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_tags": [], 2021-09-02T07:28:54.247 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2021-09-02T07:28:54.248 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2021-09-02T07:28:54.248 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2021-09-02T07:28:54.248 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2021-09-02T07:28:54.248 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2021-09-02T07:28:54.248 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2021-09-02T07:28:54.249 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:28:54.249 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2021-09-02T07:28:54.249 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2021-09-02T07:28:54.249 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2021-09-02T07:28:54.250 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2021-09-02T07:28:54.250 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2021-09-02T07:28:54.250 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2021-09-02T07:28:54.250 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:28:54.250 INFO:teuthology.orchestra.run.smithi175.stdout: "temp_url_keys": [], 2021-09-02T07:28:54.251 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "rgw", 2021-09-02T07:28:54.251 INFO:teuthology.orchestra.run.smithi175.stdout: "mfa_ids": [] 2021-09-02T07:28:54.251 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:28:54.251 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:28:54.257 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-09-02T07:28:54.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi175.front.sepia.ceph.com:8000', '--access-key', 'YDUXR4Y826KQKR5A', '--secret', 'DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:28:54.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi175.front.sepia.ceph.com:8000', '--access-key', 'YDUXR4Y826KQKR5A', '--secret', 'DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:28:54.258 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi175.front.sepia.ceph.com:8000 --access-key YDUXR4Y826KQKR5A --secret DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-02T07:28:54.309 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:28:54.309 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:28:54.408 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:28:54.408 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T07:28:54.408 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "test-realm", 2021-09-02T07:28:54.409 INFO:teuthology.orchestra.run.smithi197.stdout: "current_period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:28:54.409 INFO:teuthology.orchestra.run.smithi197.stdout: "epoch": 2 2021-09-02T07:28:54.409 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:28:54.410 INFO:tasks.rgw_multisite:creating zone.. 2021-09-02T07:28:54.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'YDUXR4Y826KQKR5A', '--secret', 'DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w', '--endpoints', 'http://smithi197.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:28:54.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'YDUXR4Y826KQKR5A', '--secret', 'DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w', '--endpoints', 'http://smithi197.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:28:54.412 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key YDUXR4Y826KQKR5A --secret DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w --endpoints http://smithi197.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-02T07:28:54.467 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:28:54.468 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:28:54.503 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:54.505+0000 7f389f21f380 0 failed reading obj info from .rgw.root:zone_info.8b6c886f-f9c4-4c5c-b256-edd3d1362f1c: (2) No such file or directory 2021-09-02T07:28:54.504 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:54.505+0000 7f389f21f380 0 WARNING: could not read zone params for zone id=8b6c886f-f9c4-4c5c-b256-edd3d1362f1c name=test-zone1 2021-09-02T07:28:54.510 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:28:54.510 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "e43b32ce-d30a-498d-8211-34e9897b4509", 2021-09-02T07:28:54.510 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "test-zone2", 2021-09-02T07:28:54.511 INFO:teuthology.orchestra.run.smithi197.stdout: "domain_root": "test-zone2.rgw.meta:root", 2021-09-02T07:28:54.511 INFO:teuthology.orchestra.run.smithi197.stdout: "control_pool": "test-zone2.rgw.control", 2021-09-02T07:28:54.511 INFO:teuthology.orchestra.run.smithi197.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2021-09-02T07:28:54.511 INFO:teuthology.orchestra.run.smithi197.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2021-09-02T07:28:54.511 INFO:teuthology.orchestra.run.smithi197.stdout: "log_pool": "test-zone2.rgw.log", 2021-09-02T07:28:54.512 INFO:teuthology.orchestra.run.smithi197.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2021-09-02T07:28:54.512 INFO:teuthology.orchestra.run.smithi197.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2021-09-02T07:28:54.512 INFO:teuthology.orchestra.run.smithi197.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2021-09-02T07:28:54.512 INFO:teuthology.orchestra.run.smithi197.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2021-09-02T07:28:54.513 INFO:teuthology.orchestra.run.smithi197.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2021-09-02T07:28:54.513 INFO:teuthology.orchestra.run.smithi197.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2021-09-02T07:28:54.513 INFO:teuthology.orchestra.run.smithi197.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2021-09-02T07:28:54.513 INFO:teuthology.orchestra.run.smithi197.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2021-09-02T07:28:54.514 INFO:teuthology.orchestra.run.smithi197.stdout: "otp_pool": "test-zone2.rgw.otp", 2021-09-02T07:28:54.514 INFO:teuthology.orchestra.run.smithi197.stdout: "system_key": { 2021-09-02T07:28:54.514 INFO:teuthology.orchestra.run.smithi197.stdout: "access_key": "YDUXR4Y826KQKR5A", 2021-09-02T07:28:54.514 INFO:teuthology.orchestra.run.smithi197.stdout: "secret_key": "DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w" 2021-09-02T07:28:54.515 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:28:54.515 INFO:teuthology.orchestra.run.smithi197.stdout: "placement_pools": [ 2021-09-02T07:28:54.515 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:28:54.515 INFO:teuthology.orchestra.run.smithi197.stdout: "key": "default-placement", 2021-09-02T07:28:54.516 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:28:54.516 INFO:teuthology.orchestra.run.smithi197.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2021-09-02T07:28:54.516 INFO:teuthology.orchestra.run.smithi197.stdout: "storage_classes": { 2021-09-02T07:28:54.516 INFO:teuthology.orchestra.run.smithi197.stdout: "STANDARD": { 2021-09-02T07:28:54.516 INFO:teuthology.orchestra.run.smithi197.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2021-09-02T07:28:54.517 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:28:54.517 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:28:54.517 INFO:teuthology.orchestra.run.smithi197.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2021-09-02T07:28:54.517 INFO:teuthology.orchestra.run.smithi197.stdout: "index_type": 0 2021-09-02T07:28:54.518 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:28:54.518 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:28:54.518 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:28:54.518 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T07:28:54.519 INFO:teuthology.orchestra.run.smithi197.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2021-09-02T07:28:54.519 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:28:54.520 DEBUG:teuthology.orchestra.run.smithi197:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2021-09-02T07:28:55.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2021-09-02T07:28:55.460 INFO:teuthology.orchestra.run.smithi197.stderr:pool 'test-zone2.rgw.buckets.data' created 2021-09-02T07:28:55.480 DEBUG:teuthology.orchestra.run.smithi197:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2021-09-02T07:28:56.496 INFO:teuthology.orchestra.run.smithi197.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2021-09-02T07:28:56.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:28:56.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:28:56.515 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-02T07:28:56.561 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:28:56.561 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:28:56.591 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:28:56.592 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "e43b32ce-d30a-498d-8211-34e9897b4509", 2021-09-02T07:28:56.592 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "test-zone2", 2021-09-02T07:28:56.592 INFO:teuthology.orchestra.run.smithi197.stdout: "domain_root": "test-zone2.rgw.meta:root", 2021-09-02T07:28:56.592 INFO:teuthology.orchestra.run.smithi197.stdout: "control_pool": "test-zone2.rgw.control", 2021-09-02T07:28:56.592 INFO:teuthology.orchestra.run.smithi197.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2021-09-02T07:28:56.593 INFO:teuthology.orchestra.run.smithi197.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2021-09-02T07:28:56.593 INFO:teuthology.orchestra.run.smithi197.stdout: "log_pool": "test-zone2.rgw.log", 2021-09-02T07:28:56.593 INFO:teuthology.orchestra.run.smithi197.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2021-09-02T07:28:56.593 INFO:teuthology.orchestra.run.smithi197.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2021-09-02T07:28:56.594 INFO:teuthology.orchestra.run.smithi197.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2021-09-02T07:28:56.594 INFO:teuthology.orchestra.run.smithi197.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2021-09-02T07:28:56.594 INFO:teuthology.orchestra.run.smithi197.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2021-09-02T07:28:56.594 INFO:teuthology.orchestra.run.smithi197.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2021-09-02T07:28:56.594 INFO:teuthology.orchestra.run.smithi197.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2021-09-02T07:28:56.595 INFO:teuthology.orchestra.run.smithi197.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2021-09-02T07:28:56.595 INFO:teuthology.orchestra.run.smithi197.stdout: "otp_pool": "test-zone2.rgw.otp", 2021-09-02T07:28:56.595 INFO:teuthology.orchestra.run.smithi197.stdout: "system_key": { 2021-09-02T07:28:56.595 INFO:teuthology.orchestra.run.smithi197.stdout: "access_key": "YDUXR4Y826KQKR5A", 2021-09-02T07:28:56.596 INFO:teuthology.orchestra.run.smithi197.stdout: "secret_key": "DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w" 2021-09-02T07:28:56.596 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:28:56.596 INFO:teuthology.orchestra.run.smithi197.stdout: "placement_pools": [ 2021-09-02T07:28:56.596 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:28:56.597 INFO:teuthology.orchestra.run.smithi197.stdout: "key": "default-placement", 2021-09-02T07:28:56.597 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:28:56.597 INFO:teuthology.orchestra.run.smithi197.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2021-09-02T07:28:56.597 INFO:teuthology.orchestra.run.smithi197.stdout: "storage_classes": { 2021-09-02T07:28:56.598 INFO:teuthology.orchestra.run.smithi197.stdout: "STANDARD": { 2021-09-02T07:28:56.598 INFO:teuthology.orchestra.run.smithi197.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2021-09-02T07:28:56.598 INFO:teuthology.orchestra.run.smithi197.stdout: "compression_type": "random" 2021-09-02T07:28:56.598 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:28:56.598 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:28:56.599 INFO:teuthology.orchestra.run.smithi197.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2021-09-02T07:28:56.599 INFO:teuthology.orchestra.run.smithi197.stdout: "index_type": 0 2021-09-02T07:28:56.599 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:28:56.599 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:28:56.600 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:28:56.600 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_id": "", 2021-09-02T07:28:56.600 INFO:teuthology.orchestra.run.smithi197.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2021-09-02T07:28:56.600 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:28:56.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:28:56.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:28:56.602 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-02T07:28:56.644 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:28:56.644 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:28:56.656 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:28:56.659+0000 7fbb8bdc9380 0 period (3c6be1ae-90c2-49d2-9d54-568c49b66f2d does not have zone e43b32ce-d30a-498d-8211-34e9897b4509 configured 2021-09-02T07:29:04.535 INFO:teuthology.orchestra.run.smithi197.stderr:Sending period to new master zone 8b6c886f-f9c4-4c5c-b256-edd3d1362f1c 2021-09-02T07:29:04.581 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:29:04.581 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:29:04.581 INFO:teuthology.orchestra.run.smithi197.stdout: "epoch": 2, 2021-09-02T07:29:04.582 INFO:teuthology.orchestra.run.smithi197.stdout: "predecessor_uuid": "dcf62b62-291a-46a6-867e-5263b7775f6a", 2021-09-02T07:29:04.582 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": [], 2021-09-02T07:29:04.582 INFO:teuthology.orchestra.run.smithi197.stdout: "period_map": { 2021-09-02T07:29:04.582 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:29:04.583 INFO:teuthology.orchestra.run.smithi197.stdout: "zonegroups": [ 2021-09-02T07:29:04.583 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:04.583 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "test-zonegroup", 2021-09-02T07:29:04.583 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "test-zonegroup", 2021-09-02T07:29:04.583 INFO:teuthology.orchestra.run.smithi197.stdout: "api_name": "", 2021-09-02T07:29:04.584 INFO:teuthology.orchestra.run.smithi197.stdout: "is_master": "true", 2021-09-02T07:29:04.584 INFO:teuthology.orchestra.run.smithi197.stdout: "endpoints": [ 2021-09-02T07:29:04.584 INFO:teuthology.orchestra.run.smithi197.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2021-09-02T07:29:04.584 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:29:04.585 INFO:teuthology.orchestra.run.smithi197.stdout: "hostnames": [], 2021-09-02T07:29:04.585 INFO:teuthology.orchestra.run.smithi197.stdout: "hostnames_s3website": [], 2021-09-02T07:29:04.585 INFO:teuthology.orchestra.run.smithi197.stdout: "master_zone": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:29:04.585 INFO:teuthology.orchestra.run.smithi197.stdout: "zones": [ 2021-09-02T07:29:04.585 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:04.586 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:29:04.586 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "test-zone1", 2021-09-02T07:29:04.586 INFO:teuthology.orchestra.run.smithi197.stdout: "endpoints": [ 2021-09-02T07:29:04.586 INFO:teuthology.orchestra.run.smithi197.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2021-09-02T07:29:04.587 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:29:04.587 INFO:teuthology.orchestra.run.smithi197.stdout: "log_meta": "false", 2021-09-02T07:29:04.587 INFO:teuthology.orchestra.run.smithi197.stdout: "log_data": "true", 2021-09-02T07:29:04.587 INFO:teuthology.orchestra.run.smithi197.stdout: "bucket_index_max_shards": 11, 2021-09-02T07:29:04.588 INFO:teuthology.orchestra.run.smithi197.stdout: "read_only": "false", 2021-09-02T07:29:04.588 INFO:teuthology.orchestra.run.smithi197.stdout: "tier_type": "", 2021-09-02T07:29:04.588 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_from_all": "true", 2021-09-02T07:29:04.588 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_from": [], 2021-09-02T07:29:04.588 INFO:teuthology.orchestra.run.smithi197.stdout: "redirect_zone": "" 2021-09-02T07:29:04.589 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:04.589 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:04.589 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "e43b32ce-d30a-498d-8211-34e9897b4509", 2021-09-02T07:29:04.589 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "test-zone2", 2021-09-02T07:29:04.590 INFO:teuthology.orchestra.run.smithi197.stdout: "endpoints": [ 2021-09-02T07:29:04.590 INFO:teuthology.orchestra.run.smithi197.stdout: "http://smithi197.front.sepia.ceph.com:8000" 2021-09-02T07:29:04.590 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:29:04.590 INFO:teuthology.orchestra.run.smithi197.stdout: "log_meta": "false", 2021-09-02T07:29:04.590 INFO:teuthology.orchestra.run.smithi197.stdout: "log_data": "true", 2021-09-02T07:29:04.591 INFO:teuthology.orchestra.run.smithi197.stdout: "bucket_index_max_shards": 11, 2021-09-02T07:29:04.591 INFO:teuthology.orchestra.run.smithi197.stdout: "read_only": "false", 2021-09-02T07:29:04.591 INFO:teuthology.orchestra.run.smithi197.stdout: "tier_type": "", 2021-09-02T07:29:04.591 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_from_all": "true", 2021-09-02T07:29:04.592 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_from": [], 2021-09-02T07:29:04.592 INFO:teuthology.orchestra.run.smithi197.stdout: "redirect_zone": "" 2021-09-02T07:29:04.592 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:04.592 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:29:04.592 INFO:teuthology.orchestra.run.smithi197.stdout: "placement_targets": [ 2021-09-02T07:29:04.593 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:04.593 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "default-placement", 2021-09-02T07:29:04.593 INFO:teuthology.orchestra.run.smithi197.stdout: "tags": [], 2021-09-02T07:29:04.593 INFO:teuthology.orchestra.run.smithi197.stdout: "storage_classes": [ 2021-09-02T07:29:04.594 INFO:teuthology.orchestra.run.smithi197.stdout: "STANDARD" 2021-09-02T07:29:04.594 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:29:04.594 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:04.594 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:29:04.595 INFO:teuthology.orchestra.run.smithi197.stdout: "default_placement": "default-placement", 2021-09-02T07:29:04.595 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T07:29:04.595 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_policy": { 2021-09-02T07:29:04.595 INFO:teuthology.orchestra.run.smithi197.stdout: "groups": [] 2021-09-02T07:29:04.595 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:04.596 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:04.596 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:29:04.596 INFO:teuthology.orchestra.run.smithi197.stdout: "short_zone_ids": [ 2021-09-02T07:29:04.596 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:04.597 INFO:teuthology.orchestra.run.smithi197.stdout: "key": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:29:04.597 INFO:teuthology.orchestra.run.smithi197.stdout: "val": 2303553998 2021-09-02T07:29:04.597 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:04.597 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:04.597 INFO:teuthology.orchestra.run.smithi197.stdout: "key": "e43b32ce-d30a-498d-8211-34e9897b4509", 2021-09-02T07:29:04.598 INFO:teuthology.orchestra.run.smithi197.stdout: "val": 1397540283 2021-09-02T07:29:04.598 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:04.598 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:29:04.598 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:04.599 INFO:teuthology.orchestra.run.smithi197.stdout: "master_zonegroup": "test-zonegroup", 2021-09-02T07:29:04.599 INFO:teuthology.orchestra.run.smithi197.stdout: "master_zone": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:29:04.599 INFO:teuthology.orchestra.run.smithi197.stdout: "period_config": { 2021-09-02T07:29:04.599 INFO:teuthology.orchestra.run.smithi197.stdout: "bucket_quota": { 2021-09-02T07:29:04.600 INFO:teuthology.orchestra.run.smithi197.stdout: "enabled": false, 2021-09-02T07:29:04.600 INFO:teuthology.orchestra.run.smithi197.stdout: "check_on_raw": false, 2021-09-02T07:29:04.600 INFO:teuthology.orchestra.run.smithi197.stdout: "max_size": -1, 2021-09-02T07:29:04.600 INFO:teuthology.orchestra.run.smithi197.stdout: "max_size_kb": 0, 2021-09-02T07:29:04.601 INFO:teuthology.orchestra.run.smithi197.stdout: "max_objects": -1 2021-09-02T07:29:04.601 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:04.602 INFO:teuthology.orchestra.run.smithi197.stdout: "user_quota": { 2021-09-02T07:29:04.602 INFO:teuthology.orchestra.run.smithi197.stdout: "enabled": false, 2021-09-02T07:29:04.603 INFO:teuthology.orchestra.run.smithi197.stdout: "check_on_raw": false, 2021-09-02T07:29:04.603 INFO:teuthology.orchestra.run.smithi197.stdout: "max_size": -1, 2021-09-02T07:29:04.603 INFO:teuthology.orchestra.run.smithi197.stdout: "max_size_kb": 0, 2021-09-02T07:29:04.603 INFO:teuthology.orchestra.run.smithi197.stdout: "max_objects": -1 2021-09-02T07:29:04.604 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:04.604 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:04.604 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T07:29:04.604 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_name": "test-realm", 2021-09-02T07:29:04.605 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:04.605 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:29:04.606 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2021-09-02T07:29:04.607 INFO:teuthology.orchestra.run:waiting for 300 2021-09-02T07:29:04.648 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:29:04.650+0000 7f3c6ba9f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 35937) UID: 0 2021-09-02T07:29:04.648 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:29:04.650+0000 7f3c7c9da440 -1 shutting down 2021-09-02T07:29:06.009 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2021-09-02T07:29:10.709 INFO:tasks.rgw.c2.client.0:Stopped 2021-09-02T07:29:10.709 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-09-02T07:29:10.710 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-09-02T07:29:10.714 INFO:tasks.rgw.c2.client.0:Started 2021-09-02T07:29:10.715 DEBUG:teuthology.orchestra.run.smithi197:> curl http://smithi197.front.sepia.ceph.com:8000 2021-09-02T07:29:10.735 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-02T07:29:10.736 INFO:teuthology.orchestra.run.smithi197.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-02T07:29:10.736 INFO:teuthology.orchestra.run.smithi197.stderr: Dload Upload Total Spent Left Speed 2021-09-02T07:29:10.737 INFO:teuthology.orchestra.run.smithi197.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi197.front.sepia.ceph.com port 8000: Connection refused 2021-09-02T07:29:11.717 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2021-09-02T07:29:11.737 DEBUG:teuthology.orchestra.run.smithi197:> curl http://smithi197.front.sepia.ceph.com:8000 2021-09-02T07:29:11.757 INFO:teuthology.orchestra.run.smithi197.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-02T07:29:11.757 INFO:teuthology.orchestra.run.smithi197.stderr: Dload Upload Total Spent Left Speed 2021-09-02T07:29:11.757 INFO:teuthology.orchestra.run.smithi197.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2021-09-02T07:29:11.758 INFO:teuthology.orchestra.run.smithi197.stdout:anonymous 2021-09-02T07:29:11.759 INFO:tasks.rgw_multisite:creating zone.. 2021-09-02T07:29:11.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'YDUXR4Y826KQKR5A', '--secret', 'DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w', '--endpoints', 'http://smithi175.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:29:11.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'YDUXR4Y826KQKR5A', '--secret', 'DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w', '--endpoints', 'http://smithi175.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:29:11.760 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key YDUXR4Y826KQKR5A --secret DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w --endpoints http://smithi175.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-02T07:29:11.814 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:29:11.814 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:29:11.847 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:29:11.849+0000 7f835b3f5380 0 failed reading obj info from .rgw.root:zone_info.e43b32ce-d30a-498d-8211-34e9897b4509: (2) No such file or directory 2021-09-02T07:29:11.847 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:29:11.849+0000 7f835b3f5380 0 WARNING: could not read zone params for zone id=e43b32ce-d30a-498d-8211-34e9897b4509 name=test-zone2 2021-09-02T07:29:11.851 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:29:11.852 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "5bdffc94-40ec-4bcf-a75a-819f7216b883", 2021-09-02T07:29:11.852 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2021-09-02T07:29:11.853 INFO:teuthology.orchestra.run.smithi175.stdout: "domain_root": "test-zone3.rgw.meta:root", 2021-09-02T07:29:11.853 INFO:teuthology.orchestra.run.smithi175.stdout: "control_pool": "test-zone3.rgw.control", 2021-09-02T07:29:11.854 INFO:teuthology.orchestra.run.smithi175.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2021-09-02T07:29:11.854 INFO:teuthology.orchestra.run.smithi175.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2021-09-02T07:29:11.855 INFO:teuthology.orchestra.run.smithi175.stdout: "log_pool": "test-zone3.rgw.log", 2021-09-02T07:29:11.855 INFO:teuthology.orchestra.run.smithi175.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2021-09-02T07:29:11.855 INFO:teuthology.orchestra.run.smithi175.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2021-09-02T07:29:11.856 INFO:teuthology.orchestra.run.smithi175.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2021-09-02T07:29:11.856 INFO:teuthology.orchestra.run.smithi175.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2021-09-02T07:29:11.857 INFO:teuthology.orchestra.run.smithi175.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2021-09-02T07:29:11.857 INFO:teuthology.orchestra.run.smithi175.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2021-09-02T07:29:11.857 INFO:teuthology.orchestra.run.smithi175.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2021-09-02T07:29:11.858 INFO:teuthology.orchestra.run.smithi175.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2021-09-02T07:29:11.858 INFO:teuthology.orchestra.run.smithi175.stdout: "otp_pool": "test-zone3.rgw.otp", 2021-09-02T07:29:11.859 INFO:teuthology.orchestra.run.smithi175.stdout: "system_key": { 2021-09-02T07:29:11.859 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "YDUXR4Y826KQKR5A", 2021-09-02T07:29:11.859 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w" 2021-09-02T07:29:11.860 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:29:11.860 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_pools": [ 2021-09-02T07:29:11.861 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:29:11.861 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "default-placement", 2021-09-02T07:29:11.862 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:29:11.862 INFO:teuthology.orchestra.run.smithi175.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2021-09-02T07:29:11.862 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": { 2021-09-02T07:29:11.863 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD": { 2021-09-02T07:29:11.863 INFO:teuthology.orchestra.run.smithi175.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2021-09-02T07:29:11.864 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:29:11.864 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:29:11.865 INFO:teuthology.orchestra.run.smithi175.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2021-09-02T07:29:11.865 INFO:teuthology.orchestra.run.smithi175.stdout: "index_type": 0 2021-09-02T07:29:11.866 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:29:11.866 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:29:11.866 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:29:11.867 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T07:29:11.867 INFO:teuthology.orchestra.run.smithi175.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2021-09-02T07:29:11.868 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:29:11.870 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2021-09-02T07:29:12.897 INFO:teuthology.orchestra.run.smithi175.stderr:pool 'test-zone3.rgw.buckets.data' created 2021-09-02T07:29:12.912 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2021-09-02T07:29:13.920 INFO:teuthology.orchestra.run.smithi175.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2021-09-02T07:29:13.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:29:13.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:29:13.931 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-02T07:29:13.976 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:29:13.977 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:29:14.005 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:29:14.005 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "5bdffc94-40ec-4bcf-a75a-819f7216b883", 2021-09-02T07:29:14.005 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2021-09-02T07:29:14.006 INFO:teuthology.orchestra.run.smithi175.stdout: "domain_root": "test-zone3.rgw.meta:root", 2021-09-02T07:29:14.006 INFO:teuthology.orchestra.run.smithi175.stdout: "control_pool": "test-zone3.rgw.control", 2021-09-02T07:29:14.006 INFO:teuthology.orchestra.run.smithi175.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2021-09-02T07:29:14.006 INFO:teuthology.orchestra.run.smithi175.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2021-09-02T07:29:14.007 INFO:teuthology.orchestra.run.smithi175.stdout: "log_pool": "test-zone3.rgw.log", 2021-09-02T07:29:14.007 INFO:teuthology.orchestra.run.smithi175.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2021-09-02T07:29:14.007 INFO:teuthology.orchestra.run.smithi175.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2021-09-02T07:29:14.007 INFO:teuthology.orchestra.run.smithi175.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2021-09-02T07:29:14.007 INFO:teuthology.orchestra.run.smithi175.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2021-09-02T07:29:14.008 INFO:teuthology.orchestra.run.smithi175.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2021-09-02T07:29:14.008 INFO:teuthology.orchestra.run.smithi175.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2021-09-02T07:29:14.008 INFO:teuthology.orchestra.run.smithi175.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2021-09-02T07:29:14.008 INFO:teuthology.orchestra.run.smithi175.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2021-09-02T07:29:14.009 INFO:teuthology.orchestra.run.smithi175.stdout: "otp_pool": "test-zone3.rgw.otp", 2021-09-02T07:29:14.009 INFO:teuthology.orchestra.run.smithi175.stdout: "system_key": { 2021-09-02T07:29:14.009 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "YDUXR4Y826KQKR5A", 2021-09-02T07:29:14.009 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w" 2021-09-02T07:29:14.009 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:29:14.010 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_pools": [ 2021-09-02T07:29:14.010 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:29:14.010 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "default-placement", 2021-09-02T07:29:14.010 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:29:14.011 INFO:teuthology.orchestra.run.smithi175.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2021-09-02T07:29:14.011 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": { 2021-09-02T07:29:14.011 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD": { 2021-09-02T07:29:14.011 INFO:teuthology.orchestra.run.smithi175.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2021-09-02T07:29:14.011 INFO:teuthology.orchestra.run.smithi175.stdout: "compression_type": "random" 2021-09-02T07:29:14.012 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:29:14.012 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:29:14.012 INFO:teuthology.orchestra.run.smithi175.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2021-09-02T07:29:14.012 INFO:teuthology.orchestra.run.smithi175.stdout: "index_type": 0 2021-09-02T07:29:14.013 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:29:14.013 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:29:14.014 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:29:14.015 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "", 2021-09-02T07:29:14.015 INFO:teuthology.orchestra.run.smithi175.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2021-09-02T07:29:14.015 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:29:14.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:29:14.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:29:14.017 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-02T07:29:14.058 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:29:14.058 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:29:14.070 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:29:14.071+0000 7f41a7e1b380 0 period (3c6be1ae-90c2-49d2-9d54-568c49b66f2d does not have zone 5bdffc94-40ec-4bcf-a75a-819f7216b883 configured 2021-09-02T07:29:21.957 INFO:teuthology.orchestra.run.smithi175.stderr:Sending period to new master zone 8b6c886f-f9c4-4c5c-b256-edd3d1362f1c 2021-09-02T07:29:21.989 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:29:21.989 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:29:21.990 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 3, 2021-09-02T07:29:21.990 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "dcf62b62-291a-46a6-867e-5263b7775f6a", 2021-09-02T07:29:21.990 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2021-09-02T07:29:21.990 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2021-09-02T07:29:21.991 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:29:21.991 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2021-09-02T07:29:21.991 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:29:21.991 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2021-09-02T07:29:21.992 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2021-09-02T07:29:21.992 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2021-09-02T07:29:21.992 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": "true", 2021-09-02T07:29:21.992 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2021-09-02T07:29:21.992 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2021-09-02T07:29:21.993 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:29:21.993 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2021-09-02T07:29:21.993 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2021-09-02T07:29:21.993 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:29:21.994 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2021-09-02T07:29:21.994 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:29:21.994 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "5bdffc94-40ec-4bcf-a75a-819f7216b883", 2021-09-02T07:29:21.994 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2021-09-02T07:29:21.995 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2021-09-02T07:29:21.995 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2021-09-02T07:29:21.995 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:29:21.995 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": "false", 2021-09-02T07:29:21.995 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": "true", 2021-09-02T07:29:21.996 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2021-09-02T07:29:21.996 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": "false", 2021-09-02T07:29:21.996 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2021-09-02T07:29:21.997 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": "true", 2021-09-02T07:29:21.997 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2021-09-02T07:29:21.997 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "" 2021-09-02T07:29:21.997 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:29:21.998 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:29:21.998 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:29:21.998 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2021-09-02T07:29:21.998 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2021-09-02T07:29:21.999 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2021-09-02T07:29:21.999 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:29:21.999 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": "false", 2021-09-02T07:29:21.999 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": "true", 2021-09-02T07:29:22.000 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2021-09-02T07:29:22.000 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": "false", 2021-09-02T07:29:22.000 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2021-09-02T07:29:22.000 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": "true", 2021-09-02T07:29:22.000 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2021-09-02T07:29:22.001 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "" 2021-09-02T07:29:22.001 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:29:22.001 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:29:22.002 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e43b32ce-d30a-498d-8211-34e9897b4509", 2021-09-02T07:29:22.002 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2021-09-02T07:29:22.002 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2021-09-02T07:29:22.002 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi197.front.sepia.ceph.com:8000" 2021-09-02T07:29:22.003 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:29:22.003 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": "false", 2021-09-02T07:29:22.003 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": "true", 2021-09-02T07:29:22.004 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2021-09-02T07:29:22.004 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": "false", 2021-09-02T07:29:22.004 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2021-09-02T07:29:22.004 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": "true", 2021-09-02T07:29:22.005 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2021-09-02T07:29:22.005 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "" 2021-09-02T07:29:22.005 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:29:22.005 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:29:22.005 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2021-09-02T07:29:22.006 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:29:22.006 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2021-09-02T07:29:22.006 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2021-09-02T07:29:22.006 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2021-09-02T07:29:22.007 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2021-09-02T07:29:22.007 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:29:22.007 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:29:22.007 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:29:22.008 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2021-09-02T07:29:22.008 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T07:29:22.008 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2021-09-02T07:29:22.008 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2021-09-02T07:29:22.008 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:29:22.009 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:29:22.009 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:29:22.009 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2021-09-02T07:29:22.009 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:29:22.010 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "5bdffc94-40ec-4bcf-a75a-819f7216b883", 2021-09-02T07:29:22.010 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 3291489215 2021-09-02T07:29:22.010 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:29:22.010 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:29:22.011 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:29:22.011 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2303553998 2021-09-02T07:29:22.011 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:29:22.012 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:29:22.012 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "e43b32ce-d30a-498d-8211-34e9897b4509", 2021-09-02T07:29:22.012 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 1397540283 2021-09-02T07:29:22.012 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:29:22.013 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:29:22.013 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:29:22.013 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2021-09-02T07:29:22.013 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:29:22.014 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2021-09-02T07:29:22.014 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2021-09-02T07:29:22.014 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2021-09-02T07:29:22.014 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2021-09-02T07:29:22.015 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2021-09-02T07:29:22.015 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2021-09-02T07:29:22.015 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2021-09-02T07:29:22.015 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:29:22.016 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2021-09-02T07:29:22.017 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2021-09-02T07:29:22.017 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2021-09-02T07:29:22.018 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2021-09-02T07:29:22.019 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2021-09-02T07:29:22.019 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2021-09-02T07:29:22.019 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:29:22.019 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:29:22.020 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T07:29:22.020 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2021-09-02T07:29:22.020 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:29:22.020 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:29:22.021 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-09-02T07:29:22.021 INFO:teuthology.orchestra.run:waiting for 300 2021-09-02T07:29:22.060 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2021-09-02T07:29:22.062+0000 7fa864fb1700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 35564) UID: 0 2021-09-02T07:29:22.060 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2021-09-02T07:29:22.062+0000 7fa875eec440 -1 shutting down 2021-09-02T07:29:23.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-09-02T07:29:28.124 INFO:tasks.rgw.c1.client.1:Stopped 2021-09-02T07:29:28.125 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-09-02T07:29:28.125 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-09-02T07:29:28.150 INFO:tasks.rgw.c1.client.1:Started 2021-09-02T07:29:28.151 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8001 2021-09-02T07:29:28.198 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-02T07:29:28.198 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2021-09-02T07:29:28.199 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi175.front.sepia.ceph.com port 8001: Connection refused 2021-09-02T07:29:28.211 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-02T07:29:29.213 DEBUG:teuthology.orchestra.run.smithi175:> curl http://smithi175.front.sepia.ceph.com:8001 2021-09-02T07:29:29.233 INFO:teuthology.orchestra.run.smithi175.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-02T07:29:29.234 INFO:teuthology.orchestra.run.smithi175.stderr: Dload Upload Total Spent Left Speed 2021-09-02T07:29:29.234 INFO:teuthology.orchestra.run.smithi175.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2021-09-02T07:29:29.236 INFO:teuthology.orchestra.run.smithi175.stdout:anonymous 2021-09-02T07:29:29.237 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-09-02T07:29:29.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi175.front.sepia.ceph.com:8000', '--access-key', 'YDUXR4Y826KQKR5A', '--secret', 'DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w', '--url', 'http://smithi175.front.sepia.ceph.com:8000', '--access-key', 'YDUXR4Y826KQKR5A', '--secret', 'DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:29:29.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi175.front.sepia.ceph.com:8000', '--access-key', 'YDUXR4Y826KQKR5A', '--secret', 'DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w', '--url', 'http://smithi175.front.sepia.ceph.com:8000', '--access-key', 'YDUXR4Y826KQKR5A', '--secret', 'DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:29:29.238 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi175.front.sepia.ceph.com:8000 --access-key YDUXR4Y826KQKR5A --secret DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w --url http://smithi175.front.sepia.ceph.com:8000 --access-key YDUXR4Y826KQKR5A --secret DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-02T07:29:29.297 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:29:29.297 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:29:29.335 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:29:29.336+0000 7f9581f7e700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.7cd628c3-b272-4627-b6df-82e8f9eb7cb4.control err=-107 2021-09-02T07:29:29.347 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:29:29.347 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T07:29:29.348 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "test-realm", 2021-09-02T07:29:29.348 INFO:teuthology.orchestra.run.smithi197.stdout: "current_period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:29:29.348 INFO:teuthology.orchestra.run.smithi197.stdout: "epoch": 2 2021-09-02T07:29:29.348 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:29:29.349 INFO:tasks.rgw_multisite:creating zone.. 2021-09-02T07:29:29.349 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'YDUXR4Y826KQKR5A', '--secret', 'DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w', '--endpoints', 'http://smithi197.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:29:29.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'YDUXR4Y826KQKR5A', '--secret', 'DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w', '--endpoints', 'http://smithi197.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:29:29.351 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key YDUXR4Y826KQKR5A --secret DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w --endpoints http://smithi197.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-02T07:29:29.405 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:29:29.405 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:29:29.434 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:29:29.435+0000 7fc643705380 0 failed reading obj info from .rgw.root:zone_info.5bdffc94-40ec-4bcf-a75a-819f7216b883: (2) No such file or directory 2021-09-02T07:29:29.434 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:29:29.435+0000 7fc643705380 0 WARNING: could not read zone params for zone id=5bdffc94-40ec-4bcf-a75a-819f7216b883 name=test-zone3 2021-09-02T07:29:29.435 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:29:29.435+0000 7fc643705380 0 failed reading obj info from .rgw.root:zone_info.8b6c886f-f9c4-4c5c-b256-edd3d1362f1c: (2) No such file or directory 2021-09-02T07:29:29.435 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:29:29.435+0000 7fc643705380 0 WARNING: could not read zone params for zone id=8b6c886f-f9c4-4c5c-b256-edd3d1362f1c name=test-zone1 2021-09-02T07:29:29.438 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:29:29.439 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "a0dd8689-bd06-4f6e-8710-337db82a68b6", 2021-09-02T07:29:29.439 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "test-zone4", 2021-09-02T07:29:29.439 INFO:teuthology.orchestra.run.smithi197.stdout: "domain_root": "test-zone4.rgw.meta:root", 2021-09-02T07:29:29.439 INFO:teuthology.orchestra.run.smithi197.stdout: "control_pool": "test-zone4.rgw.control", 2021-09-02T07:29:29.440 INFO:teuthology.orchestra.run.smithi197.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2021-09-02T07:29:29.440 INFO:teuthology.orchestra.run.smithi197.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2021-09-02T07:29:29.440 INFO:teuthology.orchestra.run.smithi197.stdout: "log_pool": "test-zone4.rgw.log", 2021-09-02T07:29:29.441 INFO:teuthology.orchestra.run.smithi197.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2021-09-02T07:29:29.441 INFO:teuthology.orchestra.run.smithi197.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2021-09-02T07:29:29.441 INFO:teuthology.orchestra.run.smithi197.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2021-09-02T07:29:29.441 INFO:teuthology.orchestra.run.smithi197.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2021-09-02T07:29:29.441 INFO:teuthology.orchestra.run.smithi197.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2021-09-02T07:29:29.442 INFO:teuthology.orchestra.run.smithi197.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2021-09-02T07:29:29.442 INFO:teuthology.orchestra.run.smithi197.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2021-09-02T07:29:29.442 INFO:teuthology.orchestra.run.smithi197.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2021-09-02T07:29:29.442 INFO:teuthology.orchestra.run.smithi197.stdout: "otp_pool": "test-zone4.rgw.otp", 2021-09-02T07:29:29.443 INFO:teuthology.orchestra.run.smithi197.stdout: "system_key": { 2021-09-02T07:29:29.443 INFO:teuthology.orchestra.run.smithi197.stdout: "access_key": "YDUXR4Y826KQKR5A", 2021-09-02T07:29:29.443 INFO:teuthology.orchestra.run.smithi197.stdout: "secret_key": "DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w" 2021-09-02T07:29:29.443 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:29.444 INFO:teuthology.orchestra.run.smithi197.stdout: "placement_pools": [ 2021-09-02T07:29:29.444 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:29.444 INFO:teuthology.orchestra.run.smithi197.stdout: "key": "default-placement", 2021-09-02T07:29:29.444 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:29:29.444 INFO:teuthology.orchestra.run.smithi197.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2021-09-02T07:29:29.445 INFO:teuthology.orchestra.run.smithi197.stdout: "storage_classes": { 2021-09-02T07:29:29.445 INFO:teuthology.orchestra.run.smithi197.stdout: "STANDARD": { 2021-09-02T07:29:29.445 INFO:teuthology.orchestra.run.smithi197.stdout: "data_pool": "test-zone4.rgw.buckets.data" 2021-09-02T07:29:29.445 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:29.446 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:29.446 INFO:teuthology.orchestra.run.smithi197.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2021-09-02T07:29:29.446 INFO:teuthology.orchestra.run.smithi197.stdout: "index_type": 0 2021-09-02T07:29:29.446 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:29.447 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:29.447 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:29:29.447 INFO:teuthology.orchestra.run.smithi197.stdout: "tier_config": { 2021-09-02T07:29:29.447 INFO:teuthology.orchestra.run.smithi197.stdout: "event_retention_days": 7, 2021-09-02T07:29:29.448 INFO:teuthology.orchestra.run.smithi197.stdout: "start_with_full_sync": false 2021-09-02T07:29:29.448 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:29.448 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T07:29:29.448 INFO:teuthology.orchestra.run.smithi197.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2021-09-02T07:29:29.449 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:29:29.450 DEBUG:teuthology.orchestra.run.smithi197:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2021-09-02T07:29:29.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-09-02T07:29:30.526 INFO:teuthology.orchestra.run.smithi197.stderr:pool 'test-zone4.rgw.buckets.data' created 2021-09-02T07:29:30.547 DEBUG:teuthology.orchestra.run.smithi197:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2021-09-02T07:29:31.547 INFO:teuthology.orchestra.run.smithi197.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2021-09-02T07:29:31.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:29:31.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:29:31.566 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-02T07:29:31.615 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:29:31.615 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:29:31.643 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:29:31.643 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "a0dd8689-bd06-4f6e-8710-337db82a68b6", 2021-09-02T07:29:31.643 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "test-zone4", 2021-09-02T07:29:31.644 INFO:teuthology.orchestra.run.smithi197.stdout: "domain_root": "test-zone4.rgw.meta:root", 2021-09-02T07:29:31.644 INFO:teuthology.orchestra.run.smithi197.stdout: "control_pool": "test-zone4.rgw.control", 2021-09-02T07:29:31.644 INFO:teuthology.orchestra.run.smithi197.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2021-09-02T07:29:31.644 INFO:teuthology.orchestra.run.smithi197.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2021-09-02T07:29:31.645 INFO:teuthology.orchestra.run.smithi197.stdout: "log_pool": "test-zone4.rgw.log", 2021-09-02T07:29:31.645 INFO:teuthology.orchestra.run.smithi197.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2021-09-02T07:29:31.645 INFO:teuthology.orchestra.run.smithi197.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2021-09-02T07:29:31.645 INFO:teuthology.orchestra.run.smithi197.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2021-09-02T07:29:31.645 INFO:teuthology.orchestra.run.smithi197.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2021-09-02T07:29:31.646 INFO:teuthology.orchestra.run.smithi197.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2021-09-02T07:29:31.646 INFO:teuthology.orchestra.run.smithi197.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2021-09-02T07:29:31.646 INFO:teuthology.orchestra.run.smithi197.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2021-09-02T07:29:31.646 INFO:teuthology.orchestra.run.smithi197.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2021-09-02T07:29:31.647 INFO:teuthology.orchestra.run.smithi197.stdout: "otp_pool": "test-zone4.rgw.otp", 2021-09-02T07:29:31.647 INFO:teuthology.orchestra.run.smithi197.stdout: "system_key": { 2021-09-02T07:29:31.647 INFO:teuthology.orchestra.run.smithi197.stdout: "access_key": "YDUXR4Y826KQKR5A", 2021-09-02T07:29:31.647 INFO:teuthology.orchestra.run.smithi197.stdout: "secret_key": "DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w" 2021-09-02T07:29:31.648 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:31.648 INFO:teuthology.orchestra.run.smithi197.stdout: "placement_pools": [ 2021-09-02T07:29:31.648 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:31.648 INFO:teuthology.orchestra.run.smithi197.stdout: "key": "default-placement", 2021-09-02T07:29:31.648 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:29:31.649 INFO:teuthology.orchestra.run.smithi197.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2021-09-02T07:29:31.649 INFO:teuthology.orchestra.run.smithi197.stdout: "storage_classes": { 2021-09-02T07:29:31.649 INFO:teuthology.orchestra.run.smithi197.stdout: "STANDARD": { 2021-09-02T07:29:31.649 INFO:teuthology.orchestra.run.smithi197.stdout: "data_pool": "test-zone4.rgw.buckets.data", 2021-09-02T07:29:31.650 INFO:teuthology.orchestra.run.smithi197.stdout: "compression_type": "random" 2021-09-02T07:29:31.650 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:31.650 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:31.650 INFO:teuthology.orchestra.run.smithi197.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2021-09-02T07:29:31.651 INFO:teuthology.orchestra.run.smithi197.stdout: "index_type": 0 2021-09-02T07:29:31.651 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:31.651 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:31.651 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:29:31.651 INFO:teuthology.orchestra.run.smithi197.stdout: "tier_config": { 2021-09-02T07:29:31.652 INFO:teuthology.orchestra.run.smithi197.stdout: "event_retention_days": 7, 2021-09-02T07:29:31.652 INFO:teuthology.orchestra.run.smithi197.stdout: "start_with_full_sync": false 2021-09-02T07:29:31.652 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:31.653 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_id": "", 2021-09-02T07:29:31.653 INFO:teuthology.orchestra.run.smithi197.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2021-09-02T07:29:31.653 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:29:31.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:29:31.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:29:31.656 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-02T07:29:31.697 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:29:31.697 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:29:31.707 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:29:31.708+0000 7f3a8387c380 0 period (3c6be1ae-90c2-49d2-9d54-568c49b66f2d does not have zone a0dd8689-bd06-4f6e-8710-337db82a68b6 configured 2021-09-02T07:29:39.601 INFO:teuthology.orchestra.run.smithi197.stderr:Sending period to new master zone 8b6c886f-f9c4-4c5c-b256-edd3d1362f1c 2021-09-02T07:29:39.641 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:29:39.641 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:29:39.641 INFO:teuthology.orchestra.run.smithi197.stdout: "epoch": 4, 2021-09-02T07:29:39.642 INFO:teuthology.orchestra.run.smithi197.stdout: "predecessor_uuid": "dcf62b62-291a-46a6-867e-5263b7775f6a", 2021-09-02T07:29:39.642 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": [], 2021-09-02T07:29:39.642 INFO:teuthology.orchestra.run.smithi197.stdout: "period_map": { 2021-09-02T07:29:39.642 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:29:39.643 INFO:teuthology.orchestra.run.smithi197.stdout: "zonegroups": [ 2021-09-02T07:29:39.643 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:39.643 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "test-zonegroup", 2021-09-02T07:29:39.643 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "test-zonegroup", 2021-09-02T07:29:39.644 INFO:teuthology.orchestra.run.smithi197.stdout: "api_name": "", 2021-09-02T07:29:39.644 INFO:teuthology.orchestra.run.smithi197.stdout: "is_master": "true", 2021-09-02T07:29:39.644 INFO:teuthology.orchestra.run.smithi197.stdout: "endpoints": [ 2021-09-02T07:29:39.644 INFO:teuthology.orchestra.run.smithi197.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2021-09-02T07:29:39.644 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:29:39.645 INFO:teuthology.orchestra.run.smithi197.stdout: "hostnames": [], 2021-09-02T07:29:39.645 INFO:teuthology.orchestra.run.smithi197.stdout: "hostnames_s3website": [], 2021-09-02T07:29:39.645 INFO:teuthology.orchestra.run.smithi197.stdout: "master_zone": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:29:39.645 INFO:teuthology.orchestra.run.smithi197.stdout: "zones": [ 2021-09-02T07:29:39.646 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:39.646 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "5bdffc94-40ec-4bcf-a75a-819f7216b883", 2021-09-02T07:29:39.646 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "test-zone3", 2021-09-02T07:29:39.647 INFO:teuthology.orchestra.run.smithi197.stdout: "endpoints": [ 2021-09-02T07:29:39.647 INFO:teuthology.orchestra.run.smithi197.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2021-09-02T07:29:39.647 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:29:39.647 INFO:teuthology.orchestra.run.smithi197.stdout: "log_meta": "false", 2021-09-02T07:29:39.648 INFO:teuthology.orchestra.run.smithi197.stdout: "log_data": "true", 2021-09-02T07:29:39.648 INFO:teuthology.orchestra.run.smithi197.stdout: "bucket_index_max_shards": 11, 2021-09-02T07:29:39.648 INFO:teuthology.orchestra.run.smithi197.stdout: "read_only": "false", 2021-09-02T07:29:39.648 INFO:teuthology.orchestra.run.smithi197.stdout: "tier_type": "", 2021-09-02T07:29:39.649 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_from_all": "true", 2021-09-02T07:29:39.649 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_from": [], 2021-09-02T07:29:39.649 INFO:teuthology.orchestra.run.smithi197.stdout: "redirect_zone": "" 2021-09-02T07:29:39.649 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:39.649 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:39.650 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:29:39.650 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "test-zone1", 2021-09-02T07:29:39.650 INFO:teuthology.orchestra.run.smithi197.stdout: "endpoints": [ 2021-09-02T07:29:39.651 INFO:teuthology.orchestra.run.smithi197.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2021-09-02T07:29:39.651 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:29:39.652 INFO:teuthology.orchestra.run.smithi197.stdout: "log_meta": "false", 2021-09-02T07:29:39.652 INFO:teuthology.orchestra.run.smithi197.stdout: "log_data": "true", 2021-09-02T07:29:39.652 INFO:teuthology.orchestra.run.smithi197.stdout: "bucket_index_max_shards": 11, 2021-09-02T07:29:39.652 INFO:teuthology.orchestra.run.smithi197.stdout: "read_only": "false", 2021-09-02T07:29:39.653 INFO:teuthology.orchestra.run.smithi197.stdout: "tier_type": "", 2021-09-02T07:29:39.653 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_from_all": "true", 2021-09-02T07:29:39.653 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_from": [], 2021-09-02T07:29:39.654 INFO:teuthology.orchestra.run.smithi197.stdout: "redirect_zone": "" 2021-09-02T07:29:39.654 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:39.654 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:39.655 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "a0dd8689-bd06-4f6e-8710-337db82a68b6", 2021-09-02T07:29:39.655 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "test-zone4", 2021-09-02T07:29:39.655 INFO:teuthology.orchestra.run.smithi197.stdout: "endpoints": [ 2021-09-02T07:29:39.655 INFO:teuthology.orchestra.run.smithi197.stdout: "http://smithi197.front.sepia.ceph.com:8001" 2021-09-02T07:29:39.655 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:29:39.656 INFO:teuthology.orchestra.run.smithi197.stdout: "log_meta": "false", 2021-09-02T07:29:39.656 INFO:teuthology.orchestra.run.smithi197.stdout: "log_data": "true", 2021-09-02T07:29:39.657 INFO:teuthology.orchestra.run.smithi197.stdout: "bucket_index_max_shards": 11, 2021-09-02T07:29:39.658 INFO:teuthology.orchestra.run.smithi197.stdout: "read_only": "false", 2021-09-02T07:29:39.658 INFO:teuthology.orchestra.run.smithi197.stdout: "tier_type": "pubsub", 2021-09-02T07:29:39.658 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_from_all": "false", 2021-09-02T07:29:39.659 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_from": [ 2021-09-02T07:29:39.659 INFO:teuthology.orchestra.run.smithi197.stdout: "test-zone1" 2021-09-02T07:29:39.659 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:29:39.659 INFO:teuthology.orchestra.run.smithi197.stdout: "redirect_zone": "" 2021-09-02T07:29:39.660 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:39.660 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:39.660 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "e43b32ce-d30a-498d-8211-34e9897b4509", 2021-09-02T07:29:39.661 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "test-zone2", 2021-09-02T07:29:39.661 INFO:teuthology.orchestra.run.smithi197.stdout: "endpoints": [ 2021-09-02T07:29:39.661 INFO:teuthology.orchestra.run.smithi197.stdout: "http://smithi197.front.sepia.ceph.com:8000" 2021-09-02T07:29:39.662 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:29:39.662 INFO:teuthology.orchestra.run.smithi197.stdout: "log_meta": "false", 2021-09-02T07:29:39.663 INFO:teuthology.orchestra.run.smithi197.stdout: "log_data": "true", 2021-09-02T07:29:39.663 INFO:teuthology.orchestra.run.smithi197.stdout: "bucket_index_max_shards": 11, 2021-09-02T07:29:39.663 INFO:teuthology.orchestra.run.smithi197.stdout: "read_only": "false", 2021-09-02T07:29:39.664 INFO:teuthology.orchestra.run.smithi197.stdout: "tier_type": "", 2021-09-02T07:29:39.664 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_from_all": "true", 2021-09-02T07:29:39.664 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_from": [], 2021-09-02T07:29:39.665 INFO:teuthology.orchestra.run.smithi197.stdout: "redirect_zone": "" 2021-09-02T07:29:39.665 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:39.665 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:29:39.666 INFO:teuthology.orchestra.run.smithi197.stdout: "placement_targets": [ 2021-09-02T07:29:39.666 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:39.667 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "default-placement", 2021-09-02T07:29:39.667 INFO:teuthology.orchestra.run.smithi197.stdout: "tags": [], 2021-09-02T07:29:39.667 INFO:teuthology.orchestra.run.smithi197.stdout: "storage_classes": [ 2021-09-02T07:29:39.668 INFO:teuthology.orchestra.run.smithi197.stdout: "STANDARD" 2021-09-02T07:29:39.668 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:29:39.669 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:39.669 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:29:39.669 INFO:teuthology.orchestra.run.smithi197.stdout: "default_placement": "default-placement", 2021-09-02T07:29:39.670 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T07:29:39.670 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_policy": { 2021-09-02T07:29:39.670 INFO:teuthology.orchestra.run.smithi197.stdout: "groups": [] 2021-09-02T07:29:39.671 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:39.671 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:39.671 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:29:39.672 INFO:teuthology.orchestra.run.smithi197.stdout: "short_zone_ids": [ 2021-09-02T07:29:39.672 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:39.674 INFO:teuthology.orchestra.run.smithi197.stdout: "key": "5bdffc94-40ec-4bcf-a75a-819f7216b883", 2021-09-02T07:29:39.675 INFO:teuthology.orchestra.run.smithi197.stdout: "val": 3291489215 2021-09-02T07:29:39.675 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:39.675 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:39.676 INFO:teuthology.orchestra.run.smithi197.stdout: "key": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:29:39.676 INFO:teuthology.orchestra.run.smithi197.stdout: "val": 2303553998 2021-09-02T07:29:39.676 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:39.677 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:39.677 INFO:teuthology.orchestra.run.smithi197.stdout: "key": "a0dd8689-bd06-4f6e-8710-337db82a68b6", 2021-09-02T07:29:39.678 INFO:teuthology.orchestra.run.smithi197.stdout: "val": 3813003288 2021-09-02T07:29:39.678 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:39.679 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:39.679 INFO:teuthology.orchestra.run.smithi197.stdout: "key": "e43b32ce-d30a-498d-8211-34e9897b4509", 2021-09-02T07:29:39.680 INFO:teuthology.orchestra.run.smithi197.stdout: "val": 1397540283 2021-09-02T07:29:39.680 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:39.681 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:29:39.681 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:39.682 INFO:teuthology.orchestra.run.smithi197.stdout: "master_zonegroup": "test-zonegroup", 2021-09-02T07:29:39.682 INFO:teuthology.orchestra.run.smithi197.stdout: "master_zone": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:29:39.682 INFO:teuthology.orchestra.run.smithi197.stdout: "period_config": { 2021-09-02T07:29:39.683 INFO:teuthology.orchestra.run.smithi197.stdout: "bucket_quota": { 2021-09-02T07:29:39.683 INFO:teuthology.orchestra.run.smithi197.stdout: "enabled": false, 2021-09-02T07:29:39.684 INFO:teuthology.orchestra.run.smithi197.stdout: "check_on_raw": false, 2021-09-02T07:29:39.684 INFO:teuthology.orchestra.run.smithi197.stdout: "max_size": -1, 2021-09-02T07:29:39.685 INFO:teuthology.orchestra.run.smithi197.stdout: "max_size_kb": 0, 2021-09-02T07:29:39.685 INFO:teuthology.orchestra.run.smithi197.stdout: "max_objects": -1 2021-09-02T07:29:39.685 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:39.686 INFO:teuthology.orchestra.run.smithi197.stdout: "user_quota": { 2021-09-02T07:29:39.686 INFO:teuthology.orchestra.run.smithi197.stdout: "enabled": false, 2021-09-02T07:29:39.687 INFO:teuthology.orchestra.run.smithi197.stdout: "check_on_raw": false, 2021-09-02T07:29:39.687 INFO:teuthology.orchestra.run.smithi197.stdout: "max_size": -1, 2021-09-02T07:29:39.688 INFO:teuthology.orchestra.run.smithi197.stdout: "max_size_kb": 0, 2021-09-02T07:29:39.688 INFO:teuthology.orchestra.run.smithi197.stdout: "max_objects": -1 2021-09-02T07:29:39.689 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:39.689 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:39.689 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T07:29:39.690 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_name": "test-realm", 2021-09-02T07:29:39.690 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:39.696 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:29:39.698 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2021-09-02T07:29:39.698 INFO:teuthology.orchestra.run:waiting for 300 2021-09-02T07:29:39.717 INFO:tasks.rgw.c2.client.1.smithi197.stdout:2021-09-02T07:29:39.719+0000 7f30149d1700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 35941) UID: 0 2021-09-02T07:29:39.717 INFO:tasks.rgw.c2.client.1.smithi197.stdout:2021-09-02T07:29:39.719+0000 7f302590c440 -1 shutting down 2021-09-02T07:29:39.782 INFO:tasks.rgw.c2.client.1:Stopped 2021-09-02T07:29:39.785 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-09-02T07:29:39.786 DEBUG:teuthology.orchestra.run.smithi197:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-09-02T07:29:39.824 INFO:tasks.rgw.c2.client.1:Started 2021-09-02T07:29:39.825 DEBUG:teuthology.orchestra.run.smithi197:> curl http://smithi197.front.sepia.ceph.com:8001 2021-09-02T07:29:39.846 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-02T07:29:39.847 INFO:teuthology.orchestra.run.smithi197.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-02T07:29:39.847 INFO:teuthology.orchestra.run.smithi197.stderr: Dload Upload Total Spent Left Speed 2021-09-02T07:29:39.848 INFO:teuthology.orchestra.run.smithi197.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi197.front.sepia.ceph.com port 8001: Connection refused 2021-09-02T07:29:40.849 DEBUG:teuthology.orchestra.run.smithi197:> curl http://smithi197.front.sepia.ceph.com:8001 2021-09-02T07:29:40.869 INFO:teuthology.orchestra.run.smithi197.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-02T07:29:40.869 INFO:teuthology.orchestra.run.smithi197.stderr: Dload Upload Total Spent Left Speed 2021-09-02T07:29:40.870 INFO:teuthology.orchestra.run.smithi197.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 130k 0 --:--:-- --:--:-- --:--:-- 130k 2021-09-02T07:29:40.871 INFO:teuthology.orchestra.run.smithi197.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-0061307d64-10e9-test-zone4","HostId":"10e9-test-zone4-test-zonegroup"} 2021-09-02T07:29:40.872 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2021-09-02T07:29:40.872 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2021-09-02T07:29:40.884 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/git.ceph.com_ceph-c_9366f375ac31713c35e559edbd160565b784db1d/qa/../src/test/rgw/rgw_multi 2021-09-02T07:29:40.903 INFO:tasks.rgw_multisite_tests:creating test user.. 2021-09-02T07:29:40.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:29:40.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:29:40.904 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-02T07:29:40.951 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:29:40.952 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:29:41.001 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:29:41.002 INFO:teuthology.orchestra.run.smithi175.stdout: "user_id": "rgw-multisite-test-user", 2021-09-02T07:29:41.002 INFO:teuthology.orchestra.run.smithi175.stdout: "display_name": "Multisite Test User", 2021-09-02T07:29:41.002 INFO:teuthology.orchestra.run.smithi175.stdout: "email": "", 2021-09-02T07:29:41.002 INFO:teuthology.orchestra.run.smithi175.stdout: "suspended": 0, 2021-09-02T07:29:41.003 INFO:teuthology.orchestra.run.smithi175.stdout: "max_buckets": 1000, 2021-09-02T07:29:41.003 INFO:teuthology.orchestra.run.smithi175.stdout: "subusers": [], 2021-09-02T07:29:41.004 INFO:teuthology.orchestra.run.smithi175.stdout: "keys": [ 2021-09-02T07:29:41.004 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:29:41.004 INFO:teuthology.orchestra.run.smithi175.stdout: "user": "rgw-multisite-test-user", 2021-09-02T07:29:41.004 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "9Q0R0363YGL2Q1WQ496K", 2021-09-02T07:29:41.005 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "1c5v4CAtROetjfZoRDfOVLDYGx61qUFxxAO8DufH" 2021-09-02T07:29:41.005 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:29:41.006 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T07:29:41.006 INFO:teuthology.orchestra.run.smithi175.stdout: "swift_keys": [], 2021-09-02T07:29:41.006 INFO:teuthology.orchestra.run.smithi175.stdout: "caps": [], 2021-09-02T07:29:41.007 INFO:teuthology.orchestra.run.smithi175.stdout: "op_mask": "read, write, delete", 2021-09-02T07:29:41.007 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "", 2021-09-02T07:29:41.007 INFO:teuthology.orchestra.run.smithi175.stdout: "default_storage_class": "", 2021-09-02T07:29:41.007 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_tags": [], 2021-09-02T07:29:41.008 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2021-09-02T07:29:41.008 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2021-09-02T07:29:41.008 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2021-09-02T07:29:41.008 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2021-09-02T07:29:41.008 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2021-09-02T07:29:41.009 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2021-09-02T07:29:41.009 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:29:41.009 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2021-09-02T07:29:41.009 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2021-09-02T07:29:41.010 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2021-09-02T07:29:41.010 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2021-09-02T07:29:41.010 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2021-09-02T07:29:41.010 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2021-09-02T07:29:41.011 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:29:41.011 INFO:teuthology.orchestra.run.smithi175.stdout: "temp_url_keys": [], 2021-09-02T07:29:41.011 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "rgw", 2021-09-02T07:29:41.011 INFO:teuthology.orchestra.run.smithi175.stdout: "mfa_ids": [] 2021-09-02T07:29:41.012 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:29:41.012 INFO:teuthology.orchestra.run.smithi175.stdout: 2021-09-02T07:29:41.016 INFO:rgw_multi.tests:meta checkpoint 2021-09-02T07:29:41.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:29:41.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:29:41.022 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:29:41.063 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:29:41.063 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:29:41.107 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T07:29:41.121 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:29:41.121 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:29:41.122 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567781.001316_39.1", 2021-09-02T07:29:41.122 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:29:41.001316Z" 2021-09-02T07:29:41.122 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:29:41.122 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:29:41.123 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:29:41.123 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:29:41.123 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:29:41.123 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:29:41.124 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:29:41.124 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:29:41.124 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:29:41.124 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:29:41.125 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:29:41.125 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:29:41.125 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:29:41.125 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:29:41.127 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T07:29:41.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:29:41.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:29:41.128 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:29:41.171 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:29:41.172 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:29:41.239 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:29:41.239 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:29:41.240 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:29:41.240 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:29:41.240 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:29:41.240 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:29:41.241 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:41.241 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:41.241 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:29:41.242 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:41.242 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:29:41.242 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:29:41.242 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:29:41.243 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:29:41.243 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:29:41.243 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:29:41.243 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:29:41.244 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:29:41.244 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:41.244 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:41.244 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:41.245 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:41.245 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:29:41.245 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:29:41.245 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:29:41.246 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:29:41.246 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:29:41.246 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:29:41.247 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:29:41.247 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:29:41.247 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:41.248 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:41.248 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:41.248 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:41.248 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:29:41.249 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:29:41.249 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:29:41.249 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:29:41.249 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:29:41.249 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:29:41.250 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:29:41.250 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:29:41.251 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:41.251 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:41.251 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:41.252 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:41.252 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:29:41.252 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:29:41.252 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:29:41.253 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:29:41.253 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:29:41.253 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:29:41.254 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:29:41.254 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:29:41.254 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:41.254 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:41.255 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:41.255 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:29:41.255 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:41.255 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:29:41.256 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:29:41.256 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:29:41.256 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:41.258 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:29:41.259 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:29:46.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:29:46.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:29:46.262 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:29:46.311 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:29:46.312 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:29:46.385 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:29:46.386 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:29:46.386 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:29:46.386 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:29:46.386 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:29:46.387 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:29:46.387 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:46.387 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:46.387 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:29:46.388 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:46.388 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:29:46.388 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:29:46.388 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:29:46.389 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:29:46.389 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:29:46.389 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:29:46.389 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:29:46.390 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:29:46.391 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:46.391 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:46.391 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:46.391 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:46.392 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:29:46.392 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:29:46.392 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:29:46.392 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:29:46.393 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:29:46.393 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:29:46.393 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:29:46.393 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:29:46.394 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:46.394 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:46.394 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:46.394 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:46.395 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:29:46.395 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:29:46.395 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:29:46.395 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:29:46.396 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:29:46.396 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:29:46.396 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:29:46.396 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:29:46.397 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:46.397 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:46.397 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:46.397 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:46.398 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:29:46.398 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:29:46.398 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:29:46.398 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:29:46.399 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:29:46.399 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:29:46.399 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:29:46.400 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:29:46.400 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:46.401 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:46.401 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:46.401 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:29:46.401 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:46.402 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:29:46.402 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:29:46.402 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:29:46.403 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:46.403 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:29:46.404 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:29:51.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:29:51.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:29:51.407 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:29:51.456 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:29:51.456 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:29:51.520 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:29:51.521 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:29:51.521 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:29:51.521 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:29:51.522 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:29:51.522 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:29:51.522 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:51.522 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:51.522 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:29:51.523 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:51.523 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:29:51.523 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:29:51.523 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:29:51.524 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:29:51.524 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:29:51.524 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:29:51.524 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:29:51.524 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:29:51.525 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:51.525 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:51.525 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:51.525 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:51.526 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:29:51.526 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:29:51.526 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:29:51.526 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:29:51.526 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:29:51.527 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:29:51.527 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:29:51.527 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:29:51.527 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:51.528 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:51.528 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:51.528 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:51.528 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:29:51.529 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:29:51.529 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:29:51.529 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:29:51.529 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:29:51.529 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:29:51.530 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:29:51.530 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:29:51.530 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:51.530 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:51.531 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:51.531 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:51.531 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:29:51.531 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:29:51.531 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:29:51.532 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:29:51.532 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:29:51.532 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:29:51.532 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:29:51.533 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:29:51.533 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:51.533 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:51.533 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:51.533 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:29:51.534 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:51.534 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:29:51.534 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:29:51.534 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:29:51.534 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:51.535 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:29:51.536 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:29:56.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:29:56.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:29:56.539 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:29:56.581 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:29:56.582 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:29:56.650 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:29:56.650 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:29:56.650 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:29:56.650 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:29:56.651 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:29:56.651 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:29:56.651 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:56.651 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:56.652 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:29:56.652 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:56.652 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:29:56.653 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:29:56.653 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:29:56.653 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:29:56.653 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:29:56.653 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:29:56.654 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:29:56.654 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:29:56.654 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:56.654 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:56.655 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:56.655 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:56.655 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:29:56.655 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:29:56.656 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:29:56.656 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:29:56.656 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:29:56.656 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:29:56.657 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:29:56.657 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:29:56.657 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:56.657 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:56.658 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:56.658 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:56.658 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:29:56.658 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:29:56.658 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:29:56.659 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:29:56.659 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:29:56.659 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:29:56.659 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:29:56.660 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:29:56.660 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:56.660 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:56.660 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:56.660 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:29:56.661 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:29:56.661 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:29:56.661 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:29:56.661 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:29:56.662 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:29:56.662 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:29:56.662 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:29:56.662 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:29:56.662 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:29:56.663 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:56.663 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:56.663 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:29:56.663 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:29:56.663 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:29:56.664 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:29:56.664 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:29:56.664 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:29:56.664 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:29:56.665 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:30:01.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:01.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:01.669 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:30:01.718 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:30:01.718 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:30:01.793 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:30:01.793 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:30:01.794 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:30:01.794 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:30:01.794 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:30:01.794 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:30:01.795 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:01.795 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:01.795 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:30:01.795 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:01.796 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:30:01.796 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:01.796 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:01.796 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:01.797 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:01.797 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:30:01.797 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:01.797 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:01.798 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:01.798 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:01.798 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:01.798 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:01.799 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:30:01.799 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:01.799 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:01.799 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:01.800 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:01.800 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:01.800 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:01.800 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:01.801 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:01.801 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:01.801 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:01.801 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:01.801 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:30:01.802 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:01.802 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:01.802 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:01.802 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:01.803 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:01.803 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:01.803 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:01.803 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:01.804 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:01.804 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:01.804 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:01.804 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:30:01.805 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:01.805 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:01.805 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:01.805 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:01.805 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:01.806 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:01.806 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:01.806 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:01.806 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:01.807 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:01.807 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:30:01.807 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:01.807 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:30:01.808 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:30:01.808 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:30:01.808 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:01.808 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:30:01.810 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:30:06.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:06.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:06.812 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:30:06.861 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:30:06.861 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:30:06.933 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:30:06.934 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:30:06.934 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:30:06.934 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:30:06.935 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:30:06.935 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:30:06.935 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:06.935 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:06.936 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:30:06.936 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:06.936 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:30:06.936 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:06.937 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:06.937 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:06.938 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:06.938 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:30:06.938 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:06.938 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:06.938 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:06.939 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:06.939 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:06.940 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:06.942 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:30:06.942 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:06.942 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:06.942 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:06.943 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:06.943 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:06.943 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:06.943 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:06.943 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:06.944 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:06.944 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:06.944 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:06.944 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:30:06.945 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:06.945 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:06.945 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:06.945 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:06.946 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:06.946 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:06.946 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:06.946 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:06.946 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:06.947 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:06.947 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:06.947 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:30:06.947 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:06.948 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:06.948 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:06.948 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:06.948 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:06.948 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:06.949 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:06.949 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:06.949 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:06.949 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:06.949 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:30:06.950 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:06.950 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:30:06.950 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:30:06.950 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:30:06.951 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:06.951 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:30:06.953 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:30:11.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:11.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:11.956 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:30:12.006 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:30:12.006 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:30:12.078 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:30:12.079 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:30:12.079 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:30:12.081 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:30:12.081 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:30:12.081 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:30:12.081 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:12.082 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:12.082 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:30:12.082 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:12.082 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:30:12.083 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:12.083 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:12.083 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:12.083 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:12.084 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:30:12.084 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:12.084 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:12.084 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:12.084 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:12.085 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:12.085 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:12.085 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:30:12.085 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:12.086 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:12.086 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:12.086 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:12.086 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:12.087 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:12.087 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:12.087 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:12.087 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:12.087 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:12.088 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:12.088 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:30:12.088 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:12.088 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:12.089 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:12.089 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:12.089 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:12.089 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:12.089 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:12.090 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:12.090 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:12.090 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:12.090 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:12.091 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:30:12.091 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:12.091 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:12.091 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:12.091 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:12.092 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:12.092 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:12.093 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:12.093 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:12.094 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:12.094 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:12.095 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:30:12.095 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:12.096 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:30:12.096 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:30:12.096 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:30:12.097 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:12.097 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:30:12.100 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:30:17.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:17.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:17.102 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:30:17.150 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:30:17.151 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:30:17.223 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:30:17.224 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:30:17.224 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:30:17.224 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:30:17.225 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:30:17.225 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:30:17.225 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:17.225 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:17.226 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:30:17.226 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:17.226 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:30:17.226 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:17.227 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:17.227 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:17.227 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:17.227 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:30:17.228 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:17.228 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:17.228 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:17.228 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:17.229 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:17.229 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:17.229 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:30:17.229 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:17.230 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:17.230 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:17.230 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:17.230 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:17.231 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:17.231 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:17.231 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:17.231 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:17.232 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:17.232 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:17.232 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:30:17.232 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:17.232 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:17.233 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:17.233 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:17.233 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:17.233 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:17.234 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:17.234 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:17.234 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:17.234 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:17.235 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:17.235 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:30:17.235 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:17.235 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:17.235 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:17.236 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:17.236 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:17.236 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:17.236 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:17.237 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:17.237 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:17.237 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:17.237 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:30:17.238 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:17.238 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:30:17.238 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:30:17.238 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:30:17.238 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:17.239 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:30:17.240 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:30:22.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:22.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:22.242 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:30:22.293 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:30:22.294 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:30:22.358 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:30:22.359 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:30:22.359 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:30:22.359 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:30:22.359 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:30:22.360 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:30:22.360 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:22.360 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:22.360 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:30:22.361 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:22.361 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:30:22.361 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:22.361 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:22.362 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:22.362 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:22.362 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:30:22.362 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:22.363 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:22.363 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:22.363 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:22.363 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:22.364 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:22.364 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:30:22.364 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:22.364 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:22.365 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:22.365 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:22.365 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:22.366 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:22.366 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:22.366 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:22.366 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:22.367 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:22.368 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:22.368 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:30:22.369 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:22.369 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:22.369 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:22.370 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:22.370 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:22.370 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:22.370 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:22.371 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:22.371 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:22.371 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:22.371 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:22.372 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:30:22.372 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:22.372 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:22.372 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:22.373 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:22.373 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:22.373 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:22.373 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:22.374 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:22.374 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:22.374 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:22.374 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:30:22.375 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:22.375 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:30:22.375 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:30:22.375 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:30:22.375 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:22.376 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:30:22.377 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:30:27.378 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:27.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:27.380 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:30:27.432 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:30:27.432 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:30:27.500 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:30:27.501 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:30:27.501 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:30:27.501 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:30:27.502 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:30:27.502 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:30:27.502 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:27.502 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:27.502 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:30:27.503 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:27.503 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:30:27.503 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:27.503 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:27.504 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:27.504 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:27.504 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:30:27.504 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:27.504 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:27.505 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:27.505 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:27.505 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:27.505 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:27.506 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:30:27.506 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:27.507 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:27.507 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:27.507 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:27.508 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:27.508 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:27.508 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:27.508 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:27.509 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:27.509 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:27.509 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:27.509 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:30:27.509 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:27.510 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:27.510 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:27.510 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:27.510 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:27.511 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:27.511 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:27.511 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:27.511 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:27.512 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:27.512 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:27.512 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:30:27.512 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:27.513 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:27.513 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:27.513 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:27.513 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:27.514 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:27.514 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:27.514 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:27.514 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:27.515 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:27.515 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:30:27.515 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:27.515 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:30:27.515 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:30:27.516 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:30:27.516 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:27.516 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:30:27.517 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:30:32.519 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:32.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:32.520 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:30:32.566 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:30:32.566 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:30:32.640 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:30:32.641 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:30:32.641 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:30:32.641 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:30:32.641 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:30:32.642 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:30:32.642 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:32.642 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:32.642 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:30:32.643 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:32.643 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:30:32.643 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:32.643 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:32.644 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:32.644 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:32.644 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:30:32.644 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:32.644 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:32.645 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:32.645 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:32.645 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:32.646 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:32.646 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:30:32.647 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:32.647 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:32.647 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:32.647 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:32.648 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:32.648 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:32.648 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:32.648 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:32.649 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:32.649 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:32.649 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:32.650 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:30:32.650 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:32.650 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:32.650 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:32.651 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:32.651 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:32.651 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:32.651 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:32.652 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:32.652 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:32.652 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:32.652 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:32.653 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:30:32.653 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:32.653 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:32.653 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:32.654 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:32.654 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:32.654 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:32.655 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:32.655 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:32.655 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:32.655 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:32.655 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:30:32.656 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:32.656 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:30:32.656 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:30:32.656 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:30:32.657 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:32.657 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:30:32.658 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:30:37.659 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:37.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:37.660 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:30:37.709 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:30:37.710 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:30:37.774 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:30:37.774 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:30:37.775 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:30:37.775 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:30:37.775 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:30:37.775 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:30:37.776 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:37.776 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:37.776 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:30:37.776 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:37.776 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:30:37.777 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:37.777 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:37.777 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:37.777 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:37.778 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:30:37.778 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:37.778 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:37.778 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:37.779 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:37.779 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:37.779 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:37.779 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:30:37.780 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:37.780 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:37.780 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:37.780 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:37.781 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:37.781 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:37.781 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:37.781 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:37.781 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:37.782 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:37.782 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:37.782 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:30:37.782 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:37.783 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:37.783 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:37.783 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:37.783 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:37.783 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:37.784 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:37.784 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:37.784 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:37.784 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:37.785 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:37.785 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:30:37.785 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:37.785 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:37.785 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:37.786 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:37.786 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:37.786 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:37.786 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:37.787 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:37.787 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:37.787 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:37.787 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:30:37.788 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:37.788 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:30:37.788 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:30:37.788 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:30:37.788 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:37.789 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:30:37.790 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:30:42.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:42.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:42.793 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:30:42.841 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:30:42.841 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:30:42.911 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:30:42.911 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:30:42.912 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:30:42.912 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:30:42.912 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:30:42.912 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:30:42.913 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:42.913 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:42.913 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:30:42.913 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:42.914 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:30:42.914 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:42.914 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:42.914 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:42.915 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:42.915 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:30:42.915 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:42.915 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:42.915 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:42.916 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:42.916 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:42.916 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:42.916 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:30:42.917 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:42.917 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:42.917 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:42.917 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:42.918 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:42.918 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:42.918 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:42.918 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:42.918 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:42.919 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:42.919 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:42.919 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:30:42.919 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:42.920 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:42.920 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:42.920 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:42.920 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:42.921 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:42.921 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:42.921 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:42.921 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:42.921 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:42.922 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:42.922 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:30:42.922 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:42.922 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:42.923 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:42.923 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:42.923 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:42.923 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:42.923 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:42.924 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:42.924 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:42.924 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:42.924 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:30:42.925 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:42.925 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:30:42.925 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:30:42.925 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:30:42.925 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:42.926 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:30:42.927 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:30:47.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:47.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:47.930 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:30:47.978 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:30:47.979 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:30:48.042 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:30:48.042 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:30:48.042 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:30:48.043 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:30:48.043 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:30:48.043 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:30:48.043 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:48.044 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:48.044 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:30:48.044 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:48.044 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:30:48.044 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:48.045 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:48.045 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:48.045 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:48.045 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:30:48.046 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:48.046 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:48.046 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:48.046 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:48.046 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:48.047 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:48.047 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:30:48.047 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:48.047 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:48.048 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:48.048 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:48.048 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:48.048 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:48.048 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:48.049 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:48.049 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:48.049 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:48.049 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:48.050 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:30:48.050 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:48.050 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:48.050 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:48.050 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:48.051 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:48.051 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:48.051 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:48.051 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:48.052 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:48.052 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:48.052 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:48.052 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:30:48.052 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:48.053 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:48.053 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:48.053 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:48.053 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:48.053 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:48.054 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:48.054 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:48.054 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:48.055 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:48.055 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:30:48.055 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:48.055 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:30:48.055 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:30:48.056 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:30:48.056 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:48.056 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:30:48.058 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:30:53.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:53.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:53.060 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:30:53.108 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:30:53.109 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:30:53.185 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:30:53.185 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:30:53.186 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:30:53.186 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:30:53.186 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:30:53.186 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:30:53.187 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:53.187 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:53.187 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:30:53.187 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:53.188 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:30:53.188 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:53.188 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:53.188 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:53.189 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:53.189 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:30:53.189 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:53.189 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:53.190 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:53.190 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:53.190 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:53.190 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:53.190 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:30:53.191 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:53.191 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:53.191 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:53.191 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:53.192 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:53.192 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:53.192 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:53.192 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:53.192 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:53.193 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:53.193 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:53.193 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:30:53.193 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:53.194 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:53.194 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:53.194 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:53.194 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:53.194 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:53.195 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:53.195 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:53.195 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:53.195 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:53.195 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:53.196 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:30:53.196 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:53.196 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:53.196 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:53.197 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:53.197 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:53.197 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:53.197 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:53.197 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:53.198 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:53.198 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:53.198 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:30:53.198 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:53.199 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:30:53.199 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:30:53.199 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:30:53.199 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:53.199 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:30:53.201 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:30:58.202 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:58.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:30:58.203 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:30:58.251 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:30:58.252 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:30:58.326 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:30:58.326 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:30:58.327 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:30:58.327 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:30:58.327 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:30:58.327 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:30:58.327 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:58.328 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:58.328 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:30:58.328 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:58.329 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:30:58.329 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:58.329 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:58.329 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:58.329 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:58.330 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:30:58.330 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:58.330 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:58.330 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:58.331 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:58.331 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:58.331 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:58.331 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:30:58.332 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:58.332 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:58.332 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:58.332 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:58.332 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:58.333 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:58.333 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:58.333 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:58.333 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:58.334 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:58.334 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:58.334 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:30:58.334 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:58.334 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:58.335 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:58.335 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:58.335 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:58.335 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:58.336 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:58.336 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:58.336 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:58.336 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:58.337 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:30:58.337 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:30:58.337 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:30:58.337 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:30:58.337 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:30:58.338 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:30:58.338 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:30:58.338 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:30:58.338 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:30:58.339 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:30:58.339 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:58.339 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:58.339 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:30:58.339 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:30:58.340 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:30:58.340 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:30:58.340 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:30:58.340 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:30:58.341 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:30:58.342 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:31:03.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:03.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:03.345 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:03.393 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:03.394 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:03.460 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:31:03.460 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:31:03.461 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:31:03.461 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:31:03.461 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:31:03.461 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:31:03.462 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:03.462 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:03.462 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:31:03.462 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:03.463 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:31:03.463 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:03.463 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:03.463 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:03.463 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:03.464 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:31:03.464 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:03.464 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:03.464 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:03.465 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:03.465 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:03.465 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:03.465 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:31:03.466 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:03.466 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:03.466 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:03.466 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:03.466 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:03.467 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:03.467 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:03.467 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:03.467 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:03.468 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:03.468 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:03.468 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:31:03.468 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:03.469 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:03.469 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:03.469 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:03.469 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:03.469 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:03.470 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:03.470 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:03.470 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:03.470 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:03.471 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:03.471 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:31:03.471 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:03.471 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:03.472 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:03.472 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:03.472 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:03.472 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:03.473 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:03.473 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:03.473 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:03.473 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:03.474 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:31:03.474 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:03.474 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:31:03.474 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:31:03.474 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:31:03.475 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:03.475 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:31:03.476 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:31:08.478 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:08.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:08.479 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:08.531 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:08.531 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:08.594 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:31:08.594 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:31:08.594 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:31:08.595 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:31:08.595 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:31:08.595 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:31:08.595 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:08.595 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:08.596 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:31:08.596 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:08.596 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:31:08.597 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:08.597 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:08.597 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:08.597 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:08.598 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:31:08.598 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:08.598 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:08.598 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:08.599 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:08.599 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:08.599 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:08.600 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:31:08.600 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:08.600 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:08.601 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:08.601 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:08.601 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:08.601 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:08.602 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:08.602 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:08.602 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:08.602 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:08.603 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:08.603 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:31:08.604 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:08.604 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:08.604 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:08.604 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:08.605 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:08.605 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:08.606 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:08.606 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:08.607 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:08.607 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:08.607 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:08.608 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:31:08.608 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:08.608 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:08.608 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:08.608 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:08.609 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:08.609 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:08.609 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:08.610 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:08.610 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:08.611 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:08.611 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:31:08.612 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:08.612 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:31:08.612 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:31:08.612 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:31:08.613 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:08.613 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:31:08.614 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:31:13.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:13.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:13.617 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:13.665 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:13.665 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:13.742 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:31:13.742 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:31:13.743 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:31:13.743 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:31:13.743 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:31:13.743 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:31:13.743 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:13.744 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:13.744 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:31:13.744 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:13.744 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:31:13.745 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:13.745 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:13.745 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:13.745 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:13.745 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:31:13.746 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:13.746 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:13.746 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:13.746 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:13.747 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:13.747 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:13.747 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:31:13.747 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:13.747 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:13.748 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:13.748 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:13.748 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:13.748 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:13.749 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:13.749 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:13.749 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:13.749 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:13.749 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:13.750 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:31:13.750 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:13.750 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:13.750 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:13.751 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:13.751 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:13.751 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:13.751 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:13.751 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:13.752 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:13.752 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:13.752 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:13.752 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:31:13.753 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:13.753 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:13.753 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:13.753 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:13.753 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:13.754 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:13.754 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:13.754 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:13.754 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:13.755 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:13.755 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:31:13.755 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:13.755 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:31:13.755 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:31:13.756 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:31:13.756 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:13.756 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:31:13.757 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:31:18.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:18.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:18.759 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:18.809 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:18.809 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:18.881 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:31:18.881 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:31:18.881 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:31:18.881 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:31:18.882 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:31:18.882 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:31:18.882 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:18.882 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:18.883 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:31:18.883 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:18.883 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:31:18.884 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:18.884 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:18.884 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:18.884 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:18.884 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:31:18.885 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:18.885 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:18.885 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:18.885 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:18.886 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:18.886 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:18.886 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:31:18.886 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:18.887 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:18.887 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:18.887 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:18.887 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:18.887 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:18.888 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:18.888 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:18.888 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:18.888 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:18.889 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:18.889 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:31:18.889 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:18.889 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:18.889 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:18.890 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:18.890 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:18.890 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:18.890 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:18.890 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:18.891 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:18.891 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:18.891 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:18.891 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:31:18.892 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:18.892 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:18.892 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:18.892 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:18.892 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:18.893 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:18.893 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:18.893 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:18.893 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:18.894 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:18.894 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:31:18.894 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:18.894 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:31:18.895 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:31:18.895 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:31:18.895 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:18.895 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:31:18.897 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:31:23.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:23.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:23.899 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:23.947 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:23.948 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:24.030 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:31:24.030 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:31:24.030 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:31:24.031 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:31:24.031 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:31:24.031 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:31:24.031 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:24.032 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:24.032 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:31:24.032 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:24.032 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:31:24.033 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:24.033 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:24.033 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:24.033 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:24.034 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:31:24.034 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:24.034 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:24.034 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:24.035 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:24.035 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:24.035 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:24.035 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:31:24.035 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:24.036 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:24.036 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:24.036 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:24.036 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:24.037 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:24.037 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:24.037 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:24.037 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:24.038 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:24.038 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:24.038 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:31:24.038 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:24.039 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:24.039 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:24.039 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:24.039 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:24.039 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:24.040 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:24.040 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:24.040 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:24.040 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:24.041 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:24.041 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:31:24.041 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:24.041 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:24.042 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:24.042 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:24.042 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:24.042 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:24.043 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:24.043 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:24.043 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:24.043 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:24.044 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:31:24.044 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:24.044 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:31:24.044 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:31:24.045 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:31:24.045 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:24.045 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:31:24.046 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:31:29.048 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:29.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:29.049 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:29.097 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:29.098 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:29.166 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:31:29.166 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:31:29.167 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:31:29.167 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:31:29.167 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:31:29.167 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:31:29.168 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:29.168 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:29.168 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:31:29.168 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:29.169 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:31:29.169 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:29.169 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:29.169 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:29.169 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:29.170 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:31:29.170 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:29.170 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:29.170 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:29.171 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:29.171 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:29.171 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:29.171 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:31:29.171 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:29.172 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:29.172 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:29.172 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:29.173 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:29.173 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:29.173 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:29.174 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:29.174 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:29.174 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:29.174 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:29.175 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:31:29.175 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:29.175 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:29.176 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:29.176 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:29.176 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:29.176 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:29.177 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:29.177 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:29.178 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:29.178 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:29.178 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:29.179 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:31:29.179 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:29.179 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:29.179 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:29.180 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:29.180 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:29.180 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:29.180 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:29.181 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:29.181 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:29.181 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:29.181 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:31:29.181 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:29.182 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:31:29.182 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:31:29.182 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:31:29.182 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:29.183 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:31:29.184 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:31:34.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:34.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:34.186 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:34.235 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:34.235 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:34.307 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:31:34.308 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:31:34.308 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:31:34.308 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:31:34.308 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:31:34.309 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:31:34.309 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:34.309 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:34.309 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:31:34.310 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:34.310 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:31:34.310 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:34.310 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:34.310 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:34.311 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:34.311 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:31:34.311 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:34.311 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:34.312 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:34.312 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:34.312 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:34.312 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:34.312 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:31:34.313 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:34.313 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:34.313 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:34.313 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:34.313 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:34.314 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:34.314 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:34.314 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:34.314 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:34.315 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:34.315 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:34.315 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:31:34.315 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:34.315 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:34.316 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:34.316 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:34.316 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:34.316 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:34.316 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:34.317 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:34.317 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:34.317 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:34.317 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:34.317 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:31:34.318 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:34.318 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:34.318 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:34.318 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:34.319 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:34.319 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:34.319 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:34.319 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:34.319 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:34.320 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:34.320 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:31:34.320 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:34.320 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:31:34.321 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:31:34.321 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:31:34.321 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:34.321 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:31:34.322 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:31:39.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:39.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:39.326 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:39.374 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:39.375 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:39.454 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:31:39.455 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:31:39.455 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:31:39.455 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:31:39.456 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:31:39.456 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:31:39.456 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:39.456 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:39.457 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:31:39.457 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:39.457 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:31:39.457 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:39.457 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:39.458 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:39.458 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:39.458 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:31:39.458 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:39.459 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:39.459 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:39.459 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:39.459 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:39.460 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:39.460 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:31:39.460 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:39.460 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:39.460 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:39.461 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:39.461 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:39.461 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:39.461 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:39.462 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:39.462 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:39.462 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:39.463 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:39.463 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:31:39.463 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:39.463 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:39.464 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:39.464 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:39.464 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:39.465 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:39.465 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:39.465 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:39.465 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:39.465 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:39.466 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:39.466 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:31:39.466 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:39.467 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:39.467 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:39.467 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:39.467 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:39.467 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:39.468 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:39.468 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:39.468 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:39.468 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:39.469 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:31:39.469 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:39.469 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:31:39.469 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:31:39.469 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:31:39.470 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:39.470 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:31:39.471 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567781.001316_39.1 target= 2021-09-02T07:31:44.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:44.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:44.474 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:44.526 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:44.526 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:44.607 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:31:44.607 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:31:44.608 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:31:44.608 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:31:44.608 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:31:44.608 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:31:44.609 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:44.609 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:44.609 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:31:44.609 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:44.610 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:31:44.610 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:44.610 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:44.610 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567781.001316_39.1", 2021-09-02T07:31:44.611 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:44.611 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:31:44.611 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:44.611 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:29:41.001316Z", 2021-09-02T07:31:44.611 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:44.612 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:44.612 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:44.612 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:44.612 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:31:44.613 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:44.613 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:44.613 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:44.613 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:44.613 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:44.614 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:44.614 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:44.614 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:44.614 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:44.615 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:44.615 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:44.615 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:31:44.615 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:44.616 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:44.616 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:44.616 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:44.616 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:44.617 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:44.617 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:44.617 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:44.617 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:44.617 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:44.618 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:44.618 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:31:44.618 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:44.618 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:44.619 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:44.619 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:44.619 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:44.619 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:44.620 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:44.620 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:44.620 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:44.620 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:44.621 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:31:44.621 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:44.621 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:31:44.621 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:31:44.621 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:31:44.622 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:44.622 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:31:44.623 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T07:31:44.624 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T07:31:44.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:44.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:44.625 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:44.674 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:44.675 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:44.742 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:31:44.742 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:31:44.743 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:31:44.743 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:31:44.743 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:31:44.743 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:31:44.744 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:31:44.744 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:44.744 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:31:44.744 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:44.745 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:31:44.745 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:31:44.745 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:31:44.745 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567781.001316_39.1", 2021-09-02T07:31:44.746 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:31:44.746 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:31:44.746 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:31:44.746 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:29:41.001316Z", 2021-09-02T07:31:44.747 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:31:44.747 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:44.747 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:44.747 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:44.747 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:31:44.748 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:31:44.748 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:31:44.748 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:31:44.748 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:31:44.749 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:31:44.749 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:31:44.749 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:31:44.749 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:31:44.750 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:44.750 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:44.750 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:44.750 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:31:44.751 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:31:44.751 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:31:44.751 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:31:44.751 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:31:44.752 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:31:44.752 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:31:44.752 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:31:44.752 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:31:44.752 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:44.753 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:44.753 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:44.755 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:31:44.755 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:31:44.755 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:31:44.755 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:31:44.756 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:31:44.757 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:31:44.757 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:31:44.757 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:31:44.758 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:31:44.758 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:44.758 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:44.759 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:31:44.759 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:44.759 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:31:44.759 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:31:44.760 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:31:44.760 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:44.760 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:31:44.761 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T07:31:44.762 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T07:31:44.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:44.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:44.763 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:44.807 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:44.807 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:44.880 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:31:44.881 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:31:44.881 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:31:44.881 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:31:44.881 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:31:44.882 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:31:44.882 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:44.882 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:44.883 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:31:44.884 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:44.885 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:31:44.886 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:44.886 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:44.886 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567781.001316_39.1", 2021-09-02T07:31:44.887 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:44.887 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:31:44.887 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:44.888 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:29:41.001316Z", 2021-09-02T07:31:44.888 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:44.888 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:44.888 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:44.889 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:44.889 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:31:44.889 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:44.889 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:44.890 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:44.890 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:44.890 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:44.890 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:44.891 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:44.891 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:44.891 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:44.891 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:44.892 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:44.892 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:31:44.892 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:44.893 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:44.893 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:44.893 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:44.893 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:44.894 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:44.894 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:44.895 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:44.895 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:44.895 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:44.895 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:44.895 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:31:44.896 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:44.896 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:44.896 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:44.896 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:44.897 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:44.897 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:44.897 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:44.897 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:44.897 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:44.898 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:44.898 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:31:44.898 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:44.898 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:31:44.899 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:31:44.899 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:31:44.899 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:44.899 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:31:44.900 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-02T07:31:44.901 INFO:rgw_multi.tests:meta checkpoint 2021-09-02T07:31:44.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:44.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:44.902 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:44.946 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:44.946 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:44.988 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T07:31:45.001 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:31:45.002 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:45.002 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567781.001316_39.1", 2021-09-02T07:31:45.002 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:29:41.001316Z" 2021-09-02T07:31:45.003 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:45.003 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:45.003 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:31:45.003 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:31:45.003 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:45.004 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:45.004 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:31:45.004 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:31:45.004 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:45.005 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:45.005 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:31:45.005 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:31:45.005 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:45.006 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:31:45.007 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T07:31:45.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:45.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:45.008 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:45.052 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:45.053 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:45.115 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:31:45.115 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:31:45.115 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:31:45.116 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:31:45.116 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:31:45.116 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:31:45.116 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:45.117 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:45.117 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:31:45.117 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:45.117 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:31:45.118 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:45.118 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:45.118 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567781.001316_39.1", 2021-09-02T07:31:45.118 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:45.118 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:31:45.119 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:45.119 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:29:41.001316Z", 2021-09-02T07:31:45.119 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:45.119 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:45.120 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:45.120 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:45.120 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:31:45.120 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:45.121 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:45.121 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:45.121 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:45.121 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:45.121 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:45.122 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:45.122 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:45.122 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:45.122 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:45.123 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:45.123 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:31:45.123 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:45.123 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:45.124 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:45.124 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:45.124 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:45.124 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:45.124 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:45.125 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:45.125 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:45.125 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:45.125 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:45.126 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:31:45.126 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:45.126 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:45.126 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:45.127 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:45.127 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:45.127 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:45.127 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:45.127 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:45.128 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:45.128 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:45.128 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:31:45.128 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:45.129 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:31:45.129 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:31:45.129 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:31:45.129 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:45.129 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:31:45.131 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T07:31:45.131 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T07:31:45.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:45.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:45.132 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:45.175 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:45.175 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:45.233 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:31:45.233 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:31:45.233 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:31:45.234 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:31:45.234 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:31:45.234 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:31:45.235 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:31:45.235 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:45.235 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:31:45.235 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:45.236 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:31:45.236 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:31:45.236 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:31:45.236 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567781.001316_39.1", 2021-09-02T07:31:45.236 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:31:45.237 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:31:45.237 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:31:45.238 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:29:41.001316Z", 2021-09-02T07:31:45.238 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:31:45.239 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:45.239 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:45.240 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:45.240 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:31:45.240 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:31:45.241 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:31:45.241 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:31:45.241 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:31:45.242 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:31:45.243 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:31:45.243 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:31:45.243 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:31:45.244 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:45.244 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:45.245 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:45.246 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:31:45.246 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:31:45.246 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:31:45.246 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:31:45.247 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:31:45.247 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:31:45.247 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:31:45.247 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:31:45.248 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:31:45.248 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:45.248 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:45.248 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:45.249 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:31:45.249 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:31:45.249 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:31:45.249 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:31:45.249 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:31:45.250 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:31:45.250 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:31:45.250 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:31:45.250 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:31:45.251 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:45.251 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:45.251 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:31:45.251 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:45.252 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:31:45.252 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:31:45.252 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:31:45.252 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:45.252 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:31:45.254 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T07:31:45.254 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T07:31:45.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:45.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:45.255 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:45.299 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:45.300 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:45.357 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:31:45.358 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:31:45.358 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:31:45.358 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:31:45.359 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:31:45.359 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:31:45.359 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:45.360 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:45.360 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:31:45.360 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:45.360 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:31:45.361 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:45.361 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:45.361 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567781.001316_39.1", 2021-09-02T07:31:45.361 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:45.362 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:31:45.362 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:45.362 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:29:41.001316Z", 2021-09-02T07:31:45.362 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:45.363 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:45.363 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:45.364 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:45.364 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:31:45.364 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:45.364 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:45.365 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:45.365 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:45.365 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:45.365 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:45.366 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:45.366 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:45.366 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:45.366 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:45.367 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:45.367 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:31:45.367 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:45.367 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:45.367 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:45.368 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:45.368 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:45.368 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:45.368 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:45.369 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:45.369 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:45.369 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:45.369 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:45.370 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:31:45.370 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:45.370 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:45.370 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:45.370 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:45.371 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:45.371 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:45.371 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:45.371 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:45.372 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:45.372 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:45.372 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:31:45.372 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:45.372 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:31:45.373 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:31:45.373 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:31:45.373 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:45.373 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:31:45.375 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-02T07:31:45.375 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/git.ceph.com_ceph-c_9366f375ac31713c35e559edbd160565b784db1d/qa/../src/test/rgw/rgw_multi' with args=['tests.py', 'tests_ps.py'] 2021-09-02T07:31:45.421 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-1 2021-09-02T07:31:48.805 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=efybpk-2 2021-09-02T07:31:52.554 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=efybpk-3 2021-09-02T07:31:52.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:52.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:52.596 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:52.641 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:52.642 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:52.687 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T07:31:52.702 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:31:52.702 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:52.703 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567912.582470_51.1", 2021-09-02T07:31:52.703 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:31:52.582470Z" 2021-09-02T07:31:52.703 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:52.703 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:52.704 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:31:52.704 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:31:52.704 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:52.704 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:52.704 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:31:52.705 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:31:52.705 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:52.705 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:52.705 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:31:52.706 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:31:52.706 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:52.706 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:31:52.707 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T07:31:52.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:52.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:52.709 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:52.752 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:52.753 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:52.817 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:31:52.818 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:31:52.818 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:31:52.818 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:31:52.818 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:31:52.819 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:31:52.819 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:52.819 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:52.819 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:31:52.820 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:52.820 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:31:52.820 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:52.820 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:52.821 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567908.795081_40.1", 2021-09-02T07:31:52.821 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:52.821 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:31:52.821 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:52.821 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:31:48.795081Z", 2021-09-02T07:31:52.822 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:52.822 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:52.822 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:52.822 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:52.823 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:31:52.823 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:52.823 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:52.823 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:52.824 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:52.824 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:52.824 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:52.824 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:52.824 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:52.825 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:52.825 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:52.825 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:52.825 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:31:52.826 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:52.826 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:52.826 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:52.826 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:52.826 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:52.827 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:52.827 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:52.828 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:52.828 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:52.828 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:52.828 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:52.829 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:31:52.829 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:52.829 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:52.829 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:52.829 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:52.830 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:52.830 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:52.830 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:52.830 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:52.831 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:52.831 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:52.831 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:31:52.831 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:52.831 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:31:52.832 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:31:52.832 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:31:52.832 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:52.832 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:31:52.834 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567912.582470_51.1 target=1_1630567908.795081_40.1 2021-09-02T07:31:57.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:57.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:57.837 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:57.886 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:57.886 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:57.968 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:31:57.968 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:31:57.969 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:31:57.969 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:31:57.970 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:31:57.970 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:31:57.971 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:57.971 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:57.971 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:31:57.972 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:57.973 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:31:57.973 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:57.974 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:57.974 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567912.582470_51.1", 2021-09-02T07:31:57.974 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:57.974 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:31:57.975 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:57.975 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:31:52.582470Z", 2021-09-02T07:31:57.975 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:57.975 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:57.976 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:57.976 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:57.976 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:31:57.976 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:57.977 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:57.977 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:57.977 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:57.977 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:57.978 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:57.978 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:57.978 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:57.979 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:57.979 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:57.979 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:57.979 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:31:57.980 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:57.980 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:57.980 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:57.980 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:57.980 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:57.981 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:57.981 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:57.981 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:57.981 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:57.982 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:57.982 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:57.983 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:31:57.983 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:57.983 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:57.983 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:57.984 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:57.984 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:57.984 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:57.984 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:57.985 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:57.985 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:57.985 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:57.985 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:31:57.985 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:57.986 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:31:57.986 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:31:57.988 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:31:57.988 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:57.988 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:31:57.989 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T07:31:57.990 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T07:31:57.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:57.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:57.992 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:58.040 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:58.041 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:58.105 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:31:58.106 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:31:58.107 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:31:58.107 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:31:58.108 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:31:58.108 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:31:58.108 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:31:58.109 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:58.109 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:31:58.110 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:58.111 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:31:58.111 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:31:58.112 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:31:58.112 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567912.582470_51.1", 2021-09-02T07:31:58.113 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:31:58.113 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:31:58.114 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:31:58.114 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:31:52.582470Z", 2021-09-02T07:31:58.115 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:31:58.115 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:58.116 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:58.116 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:58.117 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:31:58.117 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:31:58.118 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:31:58.118 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:31:58.119 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:31:58.119 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:31:58.120 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:31:58.120 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:31:58.121 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:31:58.121 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:58.122 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:58.122 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:58.122 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:31:58.123 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:31:58.123 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:31:58.123 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:31:58.124 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:31:58.124 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:31:58.124 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:31:58.124 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:31:58.125 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:31:58.125 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:58.125 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:58.126 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:58.126 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:31:58.126 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:31:58.127 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:31:58.127 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:31:58.127 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:31:58.128 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:31:58.128 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:31:58.129 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:31:58.129 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:31:58.129 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:58.130 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:58.130 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:31:58.131 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:58.131 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:31:58.131 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:31:58.132 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:31:58.132 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:58.132 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:31:58.135 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T07:31:58.135 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T07:31:58.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:58.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:58.138 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:58.181 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:58.181 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:58.240 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:31:58.240 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:31:58.240 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:31:58.241 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:31:58.241 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:31:58.241 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:31:58.241 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:58.241 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:58.242 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:31:58.242 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:58.242 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:31:58.242 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:58.243 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:58.243 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567912.582470_51.1", 2021-09-02T07:31:58.244 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:58.244 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:31:58.244 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:58.245 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:31:52.582470Z", 2021-09-02T07:31:58.246 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:58.246 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:58.247 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:58.247 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:58.248 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:31:58.248 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:58.249 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:58.249 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:58.250 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:58.250 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:58.251 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:58.251 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:58.252 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:58.253 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:58.253 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:58.254 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:58.254 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:31:58.255 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:58.255 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:58.256 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:58.256 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:58.257 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:58.257 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:58.257 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:58.258 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:58.258 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:58.258 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:58.259 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:58.259 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:31:58.259 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:58.259 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:58.260 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:58.260 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:58.260 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:58.260 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:58.261 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:58.261 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:58.261 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:58.261 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:58.262 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:31:58.262 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:58.262 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:31:58.263 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:31:58.263 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:31:58.264 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:58.264 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:31:58.266 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-02T07:31:58.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2021-09-02T07:31:58.292 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-4 2021-09-02T07:31:58.312 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=efybpk-5 2021-09-02T07:31:58.348 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=efybpk-6 2021-09-02T07:31:58.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:58.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:58.383 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:58.425 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:58.425 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:58.467 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T07:31:58.483 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:31:58.483 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:58.483 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567918.368712_63.1", 2021-09-02T07:31:58.484 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:31:58.368712Z" 2021-09-02T07:31:58.484 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:58.485 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:58.485 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:31:58.485 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:31:58.485 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:58.486 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:58.486 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:31:58.486 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:31:58.486 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:31:58.487 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:31:58.487 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:31:58.487 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:31:58.488 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:31:58.488 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:31:58.491 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T07:31:58.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:58.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:31:58.493 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:31:58.543 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:31:58.544 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:31:58.621 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:31:58.622 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:31:58.622 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:31:58.623 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:31:58.623 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:31:58.623 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:31:58.623 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:58.624 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:58.624 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:31:58.624 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:58.624 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:31:58.625 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:58.625 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:58.625 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567918.303817_52.1", 2021-09-02T07:31:58.625 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:58.625 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:31:58.626 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:58.626 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:31:58.303817Z", 2021-09-02T07:31:58.626 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:58.626 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:58.627 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:58.627 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:58.627 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:31:58.627 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:58.628 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:58.628 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:58.628 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:58.628 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:58.629 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:58.629 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:58.629 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:58.629 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:58.630 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:58.630 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:58.630 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:31:58.630 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:58.631 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:58.631 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:58.631 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:58.631 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:58.631 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:58.632 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:58.632 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:58.632 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:58.633 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:58.633 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:31:58.633 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:31:58.633 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:31:58.634 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:31:58.634 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:31:58.634 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:31:58.634 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:31:58.635 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:31:58.635 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:31:58.635 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:31:58.636 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:58.636 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:58.636 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:31:58.637 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:31:58.637 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:31:58.638 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:31:58.638 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:31:58.638 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:31:58.639 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:31:58.641 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567918.368712_63.1 target=1_1630567918.303817_52.1 2021-09-02T07:32:03.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:03.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:03.644 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:03.693 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:03.693 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:03.766 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:32:03.767 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:32:03.767 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:32:03.767 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:32:03.767 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:32:03.768 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:03.768 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:03.768 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:03.768 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:32:03.769 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:03.769 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:32:03.769 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:03.769 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:03.769 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567918.368712_63.1", 2021-09-02T07:32:03.770 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:03.770 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:32:03.770 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:03.770 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:31:58.368712Z", 2021-09-02T07:32:03.771 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:03.771 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:03.771 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:03.771 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:03.771 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:32:03.772 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:03.772 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:03.772 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:03.772 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:03.773 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:03.773 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:03.773 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:03.773 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:03.773 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:03.774 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:03.774 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:03.774 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:32:03.774 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:03.775 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:03.775 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:03.775 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:03.775 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:03.775 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:03.776 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:03.776 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:03.776 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:03.776 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:03.777 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:03.777 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:32:03.777 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:03.777 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:03.777 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:03.778 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:03.778 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:03.778 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:03.778 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:03.779 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:03.779 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:03.779 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:03.779 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:32:03.780 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:03.780 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:32:03.780 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:32:03.780 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:32:03.781 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:03.781 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:32:03.782 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T07:32:03.783 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T07:32:03.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:03.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:03.784 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:03.832 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:03.833 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:03.896 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:32:03.897 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:32:03.897 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:32:03.898 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:32:03.898 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:32:03.898 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:03.898 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:03.898 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:03.899 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:32:03.899 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:03.899 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:32:03.899 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:03.900 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:03.900 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567918.368712_63.1", 2021-09-02T07:32:03.900 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:03.900 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:32:03.900 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:03.901 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:31:58.368712Z", 2021-09-02T07:32:03.901 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:03.902 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:03.902 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:03.902 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:03.904 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:32:03.904 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:03.904 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:03.905 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:03.905 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:03.905 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:03.905 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:03.906 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:32:03.906 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:03.906 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:03.906 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:03.906 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:03.907 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:32:03.907 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:03.907 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:03.907 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:03.908 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:03.908 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:03.908 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:03.908 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:32:03.908 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:03.909 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:03.909 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:03.909 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:03.909 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:32:03.910 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:03.910 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:03.910 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:03.910 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:03.910 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:03.911 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:03.911 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:32:03.911 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:03.912 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:03.912 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:03.912 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:32:03.913 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:03.913 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:32:03.913 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:32:03.913 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:32:03.913 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:03.914 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:32:03.917 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T07:32:03.917 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T07:32:03.918 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:03.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:03.919 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:03.963 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:03.964 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:04.029 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:32:04.029 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:32:04.029 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:32:04.029 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:32:04.030 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:32:04.030 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:04.030 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:04.030 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:04.031 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:32:04.031 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:04.031 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:32:04.032 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:04.032 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:04.032 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567918.368712_63.1", 2021-09-02T07:32:04.032 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:04.033 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:32:04.033 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:04.033 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:31:58.368712Z", 2021-09-02T07:32:04.033 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:04.034 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:04.034 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:04.034 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:04.034 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:32:04.034 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:04.035 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:04.035 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:04.035 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:04.035 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:04.036 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:04.036 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:04.036 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:04.036 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:04.036 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:04.037 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:04.037 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:32:04.037 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:04.037 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:04.037 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:04.038 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:04.038 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:04.038 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:04.038 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:04.039 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:04.039 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:04.039 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:04.039 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:04.040 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:32:04.040 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:04.040 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:04.040 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:04.040 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:04.041 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:04.041 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:04.041 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:04.042 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:04.042 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:04.042 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:04.042 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:32:04.043 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:04.043 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:32:04.043 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:32:04.043 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:32:04.043 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:04.044 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:32:04.045 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-02T07:32:04.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:04.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:04.302 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:04.345 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:04.345 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:04.387 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T07:32:04.401 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:32:04.402 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:04.402 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567918.368712_63.1", 2021-09-02T07:32:04.402 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:31:58.368712Z" 2021-09-02T07:32:04.402 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:04.403 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:04.403 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:04.403 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:32:04.403 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:04.404 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:04.404 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:04.404 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:32:04.404 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:04.404 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:04.405 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:04.405 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:32:04.405 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:04.405 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:32:04.406 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T07:32:04.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:04.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:04.408 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:04.453 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:04.453 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:04.519 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:32:04.520 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:32:04.520 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:32:04.520 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:32:04.521 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:32:04.521 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:04.521 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:04.521 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:04.521 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:32:04.522 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:04.522 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:32:04.522 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:04.522 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:04.523 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567918.368712_63.1", 2021-09-02T07:32:04.523 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:04.523 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:32:04.523 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:04.523 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:31:58.368712Z", 2021-09-02T07:32:04.524 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:04.524 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:04.524 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:04.524 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:04.525 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:32:04.525 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:04.525 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:04.525 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:04.525 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:04.526 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:04.526 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:04.526 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:04.526 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:04.527 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:04.527 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:04.527 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:04.527 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:32:04.527 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:04.528 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:04.528 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:04.528 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:04.528 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:04.529 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:04.529 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:04.529 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:04.529 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:04.529 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:04.530 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:04.530 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:32:04.530 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:04.530 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:04.531 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:04.531 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:04.531 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:04.531 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:04.531 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:04.532 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:04.532 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:04.532 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:04.533 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:32:04.533 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:04.533 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:32:04.533 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:32:04.533 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:32:04.534 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:04.534 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:32:04.535 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T07:32:04.536 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T07:32:04.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:04.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:04.538 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:04.579 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:04.580 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:04.638 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:32:04.638 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:32:04.638 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:32:04.639 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:32:04.639 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:32:04.639 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:04.639 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:04.640 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:04.640 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:32:04.640 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:04.640 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:32:04.640 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:04.641 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:04.641 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567918.368712_63.1", 2021-09-02T07:32:04.641 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:04.642 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:32:04.642 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:04.642 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:31:58.368712Z", 2021-09-02T07:32:04.642 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:04.643 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:04.643 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:04.643 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:04.643 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:32:04.643 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:04.644 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:04.644 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:04.644 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:04.644 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:04.645 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:04.645 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:32:04.645 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:04.645 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:04.646 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:04.646 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:04.646 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:32:04.647 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:04.647 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:04.647 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:04.648 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:04.648 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:04.648 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:04.648 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:32:04.649 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:04.649 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:04.649 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:04.649 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:04.649 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:32:04.650 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:04.650 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:04.650 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:04.650 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:04.651 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:04.651 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:04.651 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:32:04.651 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:04.652 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:04.653 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:04.653 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:32:04.653 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:04.653 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:32:04.654 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:32:04.654 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:32:04.655 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:04.655 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:32:04.657 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T07:32:04.657 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T07:32:04.658 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:04.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:04.659 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:04.701 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:04.702 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:04.787 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:32:04.788 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:32:04.789 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:32:04.789 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:32:04.789 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:32:04.789 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:04.789 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:04.790 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:04.790 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:32:04.790 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:04.790 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:32:04.791 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:04.791 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:04.791 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567918.368712_63.1", 2021-09-02T07:32:04.791 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:04.791 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:32:04.792 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:04.792 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:31:58.368712Z", 2021-09-02T07:32:04.792 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:04.792 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:04.793 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:04.793 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:04.793 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:32:04.793 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:04.793 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:04.794 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:04.794 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:04.794 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:04.794 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:04.795 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:04.795 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:04.795 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:04.795 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:04.795 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:04.796 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:32:04.796 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:04.796 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:04.796 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:04.797 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:04.797 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:04.797 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:04.797 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:04.797 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:04.798 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:04.798 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:04.798 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:04.798 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:32:04.798 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:04.799 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:04.799 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:04.799 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:04.799 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:04.800 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:04.800 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:04.800 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:04.800 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:04.800 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:04.801 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:32:04.801 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:04.801 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:32:04.801 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:32:04.801 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:32:04.802 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:04.802 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:32:04.803 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-02T07:32:04.831 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2021-09-02T07:32:04.832 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-7 2021-09-02T07:32:04.852 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=efybpk-8 2021-09-02T07:32:04.890 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=efybpk-9 2021-09-02T07:32:04.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:04.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:04.924 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:04.966 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:04.967 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:05.010 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T07:32:05.025 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:32:05.025 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:05.026 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567924.909670_75.1", 2021-09-02T07:32:05.026 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:32:04.909670Z" 2021-09-02T07:32:05.026 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:05.026 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:05.027 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:05.030 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:32:05.031 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:05.031 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:05.031 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:05.033 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:32:05.033 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:05.034 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:05.034 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:05.034 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:32:05.035 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:05.035 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:32:05.039 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T07:32:05.039 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:05.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:05.040 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:05.087 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:05.087 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:05.147 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:32:05.147 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:32:05.148 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:32:05.148 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:32:05.148 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:32:05.148 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:05.149 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:05.149 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:05.149 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:32:05.149 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:05.149 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:32:05.150 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:05.150 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:05.150 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567924.842066_64.1", 2021-09-02T07:32:05.150 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:05.151 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:32:05.151 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:05.151 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:04.842066Z", 2021-09-02T07:32:05.151 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:05.152 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:05.152 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:05.152 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:05.152 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:32:05.153 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:05.153 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:05.153 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:05.153 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:05.154 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:05.154 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:05.154 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:05.154 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:05.155 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:05.155 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:05.155 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:05.155 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:32:05.155 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:05.156 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:05.156 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:05.156 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:05.156 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:05.157 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:05.157 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:05.157 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:05.157 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:05.157 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:05.158 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:05.158 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:32:05.158 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:05.158 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:05.159 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:05.159 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:05.159 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:05.159 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:05.160 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:05.160 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:05.160 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:05.160 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:05.160 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:32:05.161 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:05.161 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:32:05.161 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:32:05.161 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:32:05.162 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:05.162 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:32:05.163 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567924.909670_75.1 target=1_1630567924.842066_64.1 2021-09-02T07:32:10.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:10.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:10.166 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:10.215 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:10.215 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:10.288 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:32:10.288 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:32:10.288 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:32:10.289 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:32:10.289 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:32:10.289 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:10.290 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:10.290 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:10.290 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:32:10.290 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:10.291 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:32:10.291 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:10.291 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:10.291 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567924.909670_75.1", 2021-09-02T07:32:10.292 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:10.292 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:32:10.292 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:10.292 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:04.909670Z", 2021-09-02T07:32:10.292 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:10.293 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:10.293 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:10.293 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:10.293 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:32:10.294 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:10.294 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:10.294 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:10.294 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:10.295 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:10.295 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:10.295 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:10.295 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:10.295 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:10.296 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:10.296 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:10.296 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:32:10.297 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:10.297 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:10.297 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:10.297 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:10.298 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:10.298 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:10.298 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:10.298 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:10.298 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:10.299 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:10.299 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:10.299 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:32:10.300 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:10.300 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:10.300 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:10.300 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:10.301 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:10.301 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:10.301 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:10.301 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:10.302 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:10.302 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:10.302 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:32:10.302 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:10.302 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:32:10.303 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:32:10.303 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:32:10.303 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:10.303 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:32:10.304 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T07:32:10.305 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T07:32:10.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:10.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:10.307 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:10.357 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:10.357 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:10.423 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:32:10.423 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:32:10.423 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:32:10.424 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:32:10.425 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:32:10.425 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:10.426 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:10.426 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:10.426 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:32:10.427 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:10.427 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:32:10.427 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:10.427 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:10.428 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567924.909670_75.1", 2021-09-02T07:32:10.428 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:10.428 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:32:10.428 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:10.429 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:04.909670Z", 2021-09-02T07:32:10.429 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:10.430 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:10.430 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:10.430 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:10.430 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:32:10.431 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:10.431 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:10.431 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:10.431 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:10.431 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:10.432 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:10.432 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:32:10.432 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:10.432 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:10.433 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:10.433 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:10.433 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:32:10.433 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:10.434 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:10.434 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:10.434 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:10.434 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:10.434 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:10.435 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:32:10.435 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:10.435 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:10.435 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:10.436 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:10.436 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:32:10.436 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:10.436 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:10.437 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:10.437 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:10.437 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:10.437 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:10.437 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:32:10.438 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:10.438 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:10.438 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:10.438 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:32:10.439 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:10.439 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:32:10.439 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:32:10.439 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:32:10.440 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:10.440 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:32:10.443 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T07:32:10.443 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T07:32:10.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:10.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:10.446 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:10.492 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:10.492 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:10.552 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:32:10.552 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:32:10.553 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:32:10.553 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:32:10.554 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:32:10.554 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:10.554 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:10.555 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:10.555 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:32:10.556 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:10.556 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:32:10.557 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:10.557 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:10.557 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567924.909670_75.1", 2021-09-02T07:32:10.558 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:10.558 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:32:10.559 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:10.559 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:04.909670Z", 2021-09-02T07:32:10.560 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:10.560 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:10.561 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:10.561 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:10.561 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:32:10.562 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:10.562 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:10.563 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:10.563 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:10.564 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:10.564 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:10.564 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:10.565 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:10.565 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:10.566 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:10.566 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:10.567 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:32:10.567 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:10.567 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:10.568 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:10.568 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:10.569 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:10.569 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:10.569 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:10.570 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:10.570 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:10.571 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:10.571 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:10.571 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:32:10.572 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:10.572 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:10.573 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:10.573 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:10.573 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:10.574 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:10.574 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:10.575 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:10.575 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:10.575 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:10.576 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:32:10.576 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:10.577 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:32:10.577 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:32:10.577 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:32:10.578 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:10.578 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:32:10.580 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-02T07:32:10.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:10.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:10.767 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:10.812 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:10.812 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:10.855 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T07:32:10.871 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:32:10.871 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:10.872 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:32:10.872 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:32:10.734248Z" 2021-09-02T07:32:10.872 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:10.872 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:10.873 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:10.873 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:32:10.873 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:10.873 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:10.873 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:10.874 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:32:10.874 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:10.874 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:10.874 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:10.875 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:32:10.875 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:10.875 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:32:10.877 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T07:32:10.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:10.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:10.878 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:10.923 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:10.923 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:10.993 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:32:10.994 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:32:10.994 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:32:10.994 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:32:10.995 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:32:10.995 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:10.996 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:10.996 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:10.997 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:32:10.997 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:10.997 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:32:10.998 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:10.998 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:10.999 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.628530_76.1", 2021-09-02T07:32:10.999 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:11.000 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:32:11.000 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:11.000 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.628530Z", 2021-09-02T07:32:11.001 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:11.001 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:11.002 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:11.002 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:11.002 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:32:11.003 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:11.003 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:11.004 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:11.004 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:11.005 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:11.005 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:11.006 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:11.006 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:11.007 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:11.007 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:11.007 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:11.008 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:32:11.008 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:11.009 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:11.009 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:11.010 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:11.010 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:11.011 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:11.011 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:11.012 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:11.012 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:11.012 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:11.013 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:11.013 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:32:11.013 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:11.014 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:11.014 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:11.014 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:11.015 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:11.015 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:11.015 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:11.016 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:11.016 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:11.016 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:11.017 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:32:11.017 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:11.017 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:32:11.018 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:32:11.018 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:32:11.018 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:11.018 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:32:11.020 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630567930.734248_81.1 target=1_1630567930.628530_76.1 2021-09-02T07:32:16.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:16.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:16.022 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:16.070 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:16.071 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:16.145 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:32:16.146 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:32:16.146 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:32:16.146 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:32:16.146 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:32:16.147 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:16.147 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:16.147 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:16.147 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:32:16.147 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:16.148 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:32:16.148 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:16.148 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:16.148 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:32:16.149 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:16.149 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:32:16.149 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:16.149 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:32:16.149 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:16.150 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:16.150 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:16.150 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:16.150 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:32:16.151 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:16.151 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:16.151 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:16.151 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:16.152 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:16.152 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:16.152 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:16.152 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:16.152 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:16.153 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:16.153 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:16.153 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:32:16.153 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:16.154 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:16.154 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:16.154 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:16.154 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:16.155 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:16.155 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:16.155 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:16.155 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:16.155 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:16.156 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:16.156 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:32:16.156 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:16.156 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:16.157 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:16.157 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:16.157 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:16.157 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:16.157 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:16.158 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:16.158 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:16.158 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:16.158 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:32:16.159 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:16.159 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:32:16.159 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:32:16.159 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:32:16.160 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:16.160 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:32:16.161 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T07:32:16.162 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T07:32:16.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:16.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:16.163 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:16.212 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:16.213 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:16.275 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:32:16.276 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:32:16.276 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:32:16.277 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:32:16.277 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:32:16.278 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:16.279 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:16.279 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:16.279 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:32:16.279 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:16.280 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:32:16.280 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:16.280 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:16.280 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:32:16.281 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:16.281 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:32:16.281 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:16.281 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:32:16.281 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:16.282 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:16.282 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:16.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:16.282 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:32:16.283 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:16.283 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:16.283 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:16.284 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:16.284 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:16.284 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:16.284 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:32:16.285 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:16.285 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:16.285 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:16.285 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:16.285 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:32:16.286 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:16.286 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:16.286 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:16.286 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:16.287 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:16.287 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:16.287 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:32:16.287 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:16.288 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:16.288 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:16.288 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:16.288 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:32:16.288 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:16.289 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:16.289 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:16.289 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:16.289 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:16.290 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:16.290 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:32:16.290 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:16.290 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:16.291 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:16.291 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:32:16.291 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:16.291 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:32:16.291 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:32:16.292 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:32:16.292 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:16.292 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:32:16.293 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T07:32:16.294 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T07:32:16.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:16.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:16.295 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:16.340 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:16.340 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:16.411 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:32:16.412 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:32:16.412 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:32:16.412 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:32:16.413 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:32:16.413 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:16.414 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:16.414 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:16.414 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:32:16.414 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:16.414 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:32:16.415 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:16.415 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:16.415 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:32:16.415 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:16.416 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:32:16.416 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:16.416 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:32:16.416 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:16.417 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:16.417 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:16.417 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:16.417 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:32:16.417 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:16.418 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:16.418 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:16.418 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:16.419 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:16.419 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:16.419 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:16.419 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:16.420 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:16.420 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:16.420 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:16.420 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:32:16.421 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:16.421 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:16.421 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:16.421 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:16.422 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:16.422 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:16.422 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:16.422 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:16.422 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:16.423 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:16.423 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:16.423 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:32:16.424 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:16.424 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:16.424 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:16.424 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:16.425 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:16.425 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:16.425 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:16.425 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:16.426 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:16.426 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:16.427 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:32:16.427 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:16.427 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:32:16.428 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:32:16.428 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:32:16.428 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:16.428 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:32:16.430 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-02T07:32:16.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2021-09-02T07:32:16.442 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-10 2021-09-02T07:32:16.463 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=efybpk-11 2021-09-02T07:32:16.505 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=efybpk-12 2021-09-02T07:32:16.689 INFO:tasks.rgw.c2.client.1.smithi197.stdout:2021-09-02T07:32:16.690+0000 7fad65098700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:16.689 INFO:tasks.rgw.c2.client.1.smithi197.stdout:2021-09-02T07:32:16.690+0000 7fad708af700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:16.689 INFO:tasks.rgw.c2.client.1.smithi197.stdout:2021-09-02T07:32:16.690+0000 7fad64897700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:16.690 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:32:16.691+0000 7f96a35da700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:16.691 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:32:16.691+0000 7f969c5cc700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:16.691 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:32:16.691+0000 7f96a8de5700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:16.692 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2021-09-02T07:32:16.693+0000 7f0434f1e700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:16.692 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2021-09-02T07:32:16.693+0000 7f043371b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:16.692 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2021-09-02T07:32:16.693+0000 7f042df10700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:16.969 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:32:16.969+0000 7f96a25d8700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:17.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:17.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:17.136 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:17.168 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:32:17.169+0000 7f96a5ddf700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:17.169 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:32:17.170+0000 7f969f5d2700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:17.169 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:32:17.170+0000 7f969f5d2700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:17.170 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:32:17.171+0000 7f96a8de5700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:17.204 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:17.204 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:17.327 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T07:32:17.354 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:32:17.355+0000 7f969e5d0700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:17.355 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:32:17.355 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:17.355 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:32:17.355 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:32:10.734248Z" 2021-09-02T07:32:17.356 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:17.356 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:17.356 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567936.526305_41.1", 2021-09-02T07:32:17.356 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:32:16.526305Z" 2021-09-02T07:32:17.357 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:17.357 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:17.357 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:17.357 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:32:17.358 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:17.358 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:17.358 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:17.358 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:32:17.359 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:17.359 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:32:17.360 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T07:32:17.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:17.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:17.363 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:17.407 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:17.407 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:17.473 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:32:17.473 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:32:17.474 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:32:17.474 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:32:17.474 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:32:17.474 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:17.475 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:17.475 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:17.475 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:32:17.476 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:17.476 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:32:17.476 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:17.476 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:17.476 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:32:17.477 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:17.477 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:32:17.477 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:17.477 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:32:17.478 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:17.478 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:17.478 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:17.478 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:17.479 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:32:17.479 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:17.479 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:17.479 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567936.453873_30.1", 2021-09-02T07:32:17.479 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:17.480 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:17.480 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:17.480 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:16.453873Z", 2021-09-02T07:32:17.481 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:17.481 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:17.481 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:17.481 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:17.482 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:32:17.482 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:17.482 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:17.482 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:17.483 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:17.483 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:17.483 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:17.483 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:17.484 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:17.484 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:17.484 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:17.484 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:17.485 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:32:17.485 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:17.485 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:17.485 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:17.485 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:17.486 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:17.486 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:17.486 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:17.486 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:17.487 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:17.487 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:17.487 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:32:17.487 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:17.488 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:32:17.488 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:32:17.488 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:32:17.488 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:17.488 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:32:17.491 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630567936.526305_41.1 target=1_1630567936.453873_30.1 2021-09-02T07:32:22.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:22.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:22.493 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:22.555 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:22.555 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:22.654 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:32:22.655 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:32:22.655 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:32:22.655 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:32:22.655 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:32:22.656 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:22.656 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:22.656 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:22.656 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:32:22.657 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:22.657 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:32:22.657 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:22.658 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:22.658 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:32:22.659 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:22.659 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:32:22.660 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:22.660 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:32:22.660 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:22.661 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:22.661 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:22.661 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:22.662 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:32:22.662 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:22.662 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:22.663 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567936.526305_41.1", 2021-09-02T07:32:22.663 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:22.663 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:22.664 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:22.664 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:16.526305Z", 2021-09-02T07:32:22.664 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:22.664 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:22.664 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:22.665 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:22.665 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:32:22.665 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:22.665 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:22.666 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:22.666 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:22.666 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:22.666 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:22.667 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:22.667 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:22.667 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:22.667 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:22.668 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:22.668 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:32:22.668 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:22.668 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:22.669 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:22.669 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:22.669 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:22.670 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:22.670 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:22.671 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:22.671 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:22.672 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:22.672 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:32:22.672 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:22.672 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:32:22.673 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:32:22.673 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:32:22.673 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:22.673 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:32:22.674 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T07:32:22.675 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T07:32:22.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:22.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:22.676 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:22.723 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:22.723 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:22.807 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:32:22.808 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:32:22.808 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:32:22.808 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:32:22.809 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:32:22.809 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:22.809 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:22.809 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:22.810 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:32:22.810 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:22.810 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:32:22.810 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:22.811 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:22.811 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:32:22.811 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:22.811 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:32:22.811 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:22.812 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:32:22.812 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:22.812 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:22.812 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:22.813 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:22.813 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:32:22.813 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:22.813 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:22.813 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567936.453873_30.1", 2021-09-02T07:32:22.814 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:22.814 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:22.814 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:22.814 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:16.453873Z", 2021-09-02T07:32:22.815 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:22.815 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:22.815 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:22.815 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:22.816 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:32:22.816 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:22.816 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:22.816 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:22.817 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:22.817 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:22.817 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:22.817 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:32:22.818 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:22.818 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:22.818 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:22.818 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:22.819 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:32:22.819 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:22.819 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:22.819 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:22.819 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:22.820 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:22.820 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:22.820 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:32:22.820 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:22.821 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:22.821 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:22.821 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:32:22.821 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:22.821 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:32:22.822 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:32:22.822 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:32:22.822 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:22.822 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:32:22.824 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630567936.526305_41.1 target=1_1630567936.453873_30.1 2021-09-02T07:32:27.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:27.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:27.826 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:27.875 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:27.875 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:27.954 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:32:27.955 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:32:27.955 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:32:27.955 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:32:27.956 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:32:27.956 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:27.956 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:27.956 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:27.957 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:32:27.957 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:27.957 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:32:27.957 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:27.958 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:27.958 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:32:27.958 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:27.958 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:32:27.959 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:27.959 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:32:27.959 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:27.959 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:27.960 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:27.960 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:27.960 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:32:27.961 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:27.962 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:27.963 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567936.526305_41.1", 2021-09-02T07:32:27.963 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:27.964 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:27.964 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:27.965 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:16.526305Z", 2021-09-02T07:32:27.965 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:27.965 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:27.965 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:27.966 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:27.966 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:32:27.966 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:27.966 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:27.967 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:27.967 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:27.967 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:27.967 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:27.967 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:32:27.968 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:27.968 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:27.968 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:27.968 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:27.969 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:32:27.969 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:27.969 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:27.969 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:27.969 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:27.970 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:27.970 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:27.970 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:32:27.970 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:27.971 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:27.971 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:27.971 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:32:27.971 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:27.972 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:32:27.972 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:32:27.972 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:32:27.972 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:27.972 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:32:27.974 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T07:32:27.974 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T07:32:27.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:27.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:27.976 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:28.024 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:28.025 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:28.093 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:32:28.094 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:32:28.094 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:32:28.094 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:32:28.095 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:32:28.095 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:28.095 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:28.095 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:28.096 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:32:28.096 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:28.096 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:32:28.096 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:28.096 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:28.097 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:32:28.097 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:28.097 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:32:28.097 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:28.098 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:32:28.098 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:28.098 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:28.098 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:28.099 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:28.099 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:32:28.099 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:28.099 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:28.099 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567936.526305_41.1", 2021-09-02T07:32:28.100 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:28.100 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:28.100 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:28.100 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:16.526305Z", 2021-09-02T07:32:28.100 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:28.101 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:28.102 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:28.103 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:28.103 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:32:28.103 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:28.104 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:28.104 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:28.104 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:28.105 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:28.105 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:28.106 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:28.106 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:28.106 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:28.106 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:28.107 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:28.107 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:32:28.107 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:28.107 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:28.107 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:28.108 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:28.108 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:28.108 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:28.108 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:28.109 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:28.109 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:28.109 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:28.109 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:32:28.110 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:28.110 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:32:28.110 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:32:28.110 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:32:28.110 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:28.111 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:32:28.111 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-02T07:32:28.112 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:28.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:28.113 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:32:28.162 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:28.163 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:28.184 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:28.186+0000 7f52cc20d380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:32:28.185 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:28.186+0000 7f52cc20d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:32:28.185 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:28.186+0000 7f52cc20d380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:32:28.185 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:28.186+0000 7f52cc20d380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:32:28.216 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:28.218+0000 7f52cc20d380 1 bucket sync caught up with source: 2021-09-02T07:32:28.217 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000001.8.5, , 00000000001.5.5, , , , 00000000001.8.5, 00000000001.7.5, , ] 2021-09-02T07:32:28.217 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000001.8.5, , 00000000001.5.5, , , , 00000000001.8.5, 00000000001.7.5, , ] 2021-09-02T07:32:28.217 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:28.218+0000 7f52cc20d380 0 bucket checkpoint complete 2021-09-02T07:32:28.234 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-10 zones={test-zone2, test-zone1} 2021-09-02T07:32:28.241 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:32:28.246 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-02T07:32:28.246 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-02T07:32:28.247 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-02T07:32:28.247 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:28.248 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:32:28.251 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-02T07:32:28.252 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-02T07:32:28.252 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-02T07:32:28.253 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:28.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-02T07:32:28.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-02T07:32:28.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-02T07:32:28.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-02T07:32:28.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-02T07:32:28.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-02T07:32:28.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-02T07:32:28.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-02T07:32:28.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-02T07:32:29.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:29.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:29.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:29.286 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-10 zones={test-zone2, test-zone1} 2021-09-02T07:32:29.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:29.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:29.288 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:32:29.343 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:29.344 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:29.368 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:29.369+0000 7fe15a259380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:32:29.368 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:29.369+0000 7fe15a259380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:32:29.368 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:29.369+0000 7fe15a259380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:32:29.369 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:29.369+0000 7fe15a259380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:32:29.400 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:29.402+0000 7fe15a259380 1 bucket sync caught up with source: 2021-09-02T07:32:29.401 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.8.5, , 00000000001.5.5, , , , 00000000001.8.5, 00000000001.7.5, , ] 2021-09-02T07:32:29.401 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000001.8.5, , 00000000001.5.5, , , , 00000000001.8.5, 00000000001.7.5, , ] 2021-09-02T07:32:29.401 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:29.402+0000 7fe15a259380 0 bucket checkpoint complete 2021-09-02T07:32:29.416 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-10 zones={test-zone3, test-zone1} 2021-09-02T07:32:29.422 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:32:29.427 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-02T07:32:29.427 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-02T07:32:29.428 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-02T07:32:29.429 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:29.429 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:32:29.433 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-02T07:32:29.433 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-02T07:32:29.434 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-02T07:32:29.434 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:29.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-02T07:32:29.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-02T07:32:29.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-02T07:32:29.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-02T07:32:29.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-02T07:32:29.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-02T07:32:29.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-02T07:32:30.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-02T07:32:30.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-02T07:32:30.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:30.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:30.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:30.458 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-10 zones={test-zone3, test-zone1} 2021-09-02T07:32:30.459 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:30.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:30.460 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:32:30.518 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:30.519 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:30.551 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:30.552+0000 7f02bc68b380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:32:30.552 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:30.552+0000 7f02bc68b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:32:30.552 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:30.552+0000 7f02bc68b380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:32:30.552 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:30.552+0000 7f02bc68b380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:32:30.593 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:30.594+0000 7f02bc68b380 1 bucket sync caught up with source: 2021-09-02T07:32:30.593 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000001.8.5, , 00000000001.5.5, , , , 00000000001.8.5, 00000000001.7.5, , ] 2021-09-02T07:32:30.594 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000001.8.5, , 00000000001.5.5, , , , 00000000001.8.5, 00000000001.7.5, , ] 2021-09-02T07:32:30.594 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:30.594+0000 7f02bc68b380 0 bucket checkpoint complete 2021-09-02T07:32:30.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:30.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:30.609 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:32:30.660 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:30.661 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:30.683 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:30.684+0000 7f8d4f65d380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:32:30.683 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:30.684+0000 7f8d4f65d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:32:30.684 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:30.684+0000 7f8d4f65d380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:32:30.684 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:30.684+0000 7f8d4f65d380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:32:30.715 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:30.716+0000 7f8d4f65d380 1 bucket sync caught up with source: 2021-09-02T07:32:30.715 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.72.5, , 00000000001.14.5, , , , 00000000001.25.5, 00000000001.2.5, , ] 2021-09-02T07:32:30.715 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000001.72.5, , 00000000001.14.5, , , , 00000000001.25.5, 00000000001.2.5, , ] 2021-09-02T07:32:30.716 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:30.716+0000 7f8d4f65d380 0 bucket checkpoint complete 2021-09-02T07:32:30.729 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-11 zones={test-zone1, test-zone2} 2021-09-02T07:32:30.735 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:32:30.738 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-02T07:32:30.739 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-02T07:32:30.739 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-02T07:32:30.740 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:30.740 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:32:30.743 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-02T07:32:30.744 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-02T07:32:30.744 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-02T07:32:30.745 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:30.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-02T07:32:30.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-02T07:32:30.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-02T07:32:31.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-02T07:32:31.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-02T07:32:31.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-02T07:32:31.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-02T07:32:31.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-02T07:32:31.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-02T07:32:31.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:31.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:31.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:31.819 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-11 zones={test-zone1, test-zone2} 2021-09-02T07:32:31.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:31.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:31.820 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:32:31.872 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:31.873 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:31.895 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:31.897+0000 7f1b6de24380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:32:31.896 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:31.897+0000 7f1b6de24380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:32:31.896 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:31.897+0000 7f1b6de24380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:32:31.896 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:31.897+0000 7f1b6de24380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:32:31.940 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:31.941+0000 7f1b6de24380 1 bucket sync caught up with source: 2021-09-02T07:32:31.940 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.72.5, , 00000000001.14.5, , , , 00000000001.25.5, 00000000001.2.5, , ] 2021-09-02T07:32:31.941 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000001.72.5, , 00000000001.14.5, , , , 00000000001.25.5, 00000000001.2.5, , ] 2021-09-02T07:32:31.941 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:31.941+0000 7f1b6de24380 0 bucket checkpoint complete 2021-09-02T07:32:31.958 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-11 zones={test-zone3, test-zone2} 2021-09-02T07:32:31.965 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:32:31.968 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-02T07:32:31.969 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-02T07:32:31.969 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-02T07:32:31.970 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:31.970 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:32:32.015 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-02T07:32:32.016 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-02T07:32:32.016 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-02T07:32:32.017 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:32.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-02T07:32:32.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-02T07:32:32.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-02T07:32:32.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-02T07:32:32.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-02T07:32:32.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-02T07:32:32.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-02T07:32:32.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-02T07:32:32.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-02T07:32:32.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:32.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:32.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:33.050 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-11 zones={test-zone3, test-zone2} 2021-09-02T07:32:33.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:33.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:33.052 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:32:33.105 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:33.105 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:33.128 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:33.130+0000 7f0f42b59380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:32:33.129 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:33.130+0000 7f0f42b59380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:32:33.129 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:33.130+0000 7f0f42b59380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:32:33.129 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:33.130+0000 7f0f42b59380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:32:33.167 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:33.168+0000 7f0f42b59380 1 bucket sync caught up with source: 2021-09-02T07:32:33.167 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.12.5, , 00000000001.61.5, , , , 00000000001.44.5, 00000000001.23.5, , ] 2021-09-02T07:32:33.167 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000001.12.5, , 00000000001.61.5, , , , 00000000001.44.5, 00000000001.23.5, , ] 2021-09-02T07:32:33.168 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:33.168+0000 7f0f42b59380 0 bucket checkpoint complete 2021-09-02T07:32:33.184 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-12 zones={test-zone1, test-zone3} 2021-09-02T07:32:33.190 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:32:33.193 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-02T07:32:33.194 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-02T07:32:33.194 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-02T07:32:33.195 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:33.195 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:32:33.199 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-02T07:32:33.199 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-02T07:32:33.200 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-02T07:32:33.201 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:33.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-02T07:32:33.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-02T07:32:33.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-02T07:32:33.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-02T07:32:33.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-02T07:32:33.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-02T07:32:33.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-02T07:32:33.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-02T07:32:33.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-02T07:32:33.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:34.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:34.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:34.226 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-12 zones={test-zone1, test-zone3} 2021-09-02T07:32:34.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:34.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:34.230 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:32:34.287 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:34.287 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:34.313 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:34.315+0000 7f4650f34380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:32:34.313 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:34.315+0000 7f4650f34380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:32:34.314 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:34.315+0000 7f4650f34380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:32:34.314 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:34.315+0000 7f4650f34380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:32:34.350 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:34.352+0000 7f4650f34380 1 bucket sync caught up with source: 2021-09-02T07:32:34.350 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000001.12.5, , 00000000001.61.5, , , , 00000000001.44.5, 00000000001.23.5, , ] 2021-09-02T07:32:34.350 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000001.12.5, , 00000000001.61.5, , , , 00000000001.44.5, 00000000001.23.5, , ] 2021-09-02T07:32:34.351 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:34.352+0000 7f4650f34380 0 bucket checkpoint complete 2021-09-02T07:32:34.365 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-12 zones={test-zone2, test-zone3} 2021-09-02T07:32:34.373 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:32:34.376 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-02T07:32:34.377 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-02T07:32:34.377 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-02T07:32:34.378 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:34.378 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:32:34.381 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-02T07:32:34.382 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-02T07:32:34.382 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-02T07:32:34.383 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:34.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-02T07:32:34.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-02T07:32:34.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-02T07:32:34.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-02T07:32:34.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-02T07:32:34.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-02T07:32:34.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-02T07:32:34.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-02T07:32:35.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-02T07:32:35.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:35.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:35.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:35.414 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-12 zones={test-zone2, test-zone3} 2021-09-02T07:32:35.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2021-09-02T07:32:35.416 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-13 2021-09-02T07:32:35.443 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=efybpk-14 2021-09-02T07:32:35.478 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=efybpk-15 2021-09-02T07:32:35.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:35.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:35.664 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:35.706 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2021-09-02T07:32:35.686+0000 7f0433f1c700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:35.708 INFO:tasks.rgw.c2.client.1.smithi197.stdout:2021-09-02T07:32:35.710+0000 7fad710b0700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:35.708 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:32:35.710+0000 7f96a4ddd700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:35.709 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2021-09-02T07:32:35.710+0000 7f0434f1e700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:35.709 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:32:35.712+0000 7f969fdd3700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:35.710 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:35.710 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:35.711 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2021-09-02T07:32:35.712+0000 7f042ff14700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:35.774 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T07:32:35.776 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:32:35.779+0000 7f969ddcf700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:32:35.791 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:32:35.792 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:35.792 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:32:35.792 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:32:10.734248Z" 2021-09-02T07:32:35.793 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:35.793 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:35.793 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567955.501542_59.1", 2021-09-02T07:32:35.793 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:32:35.501542Z" 2021-09-02T07:32:35.793 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:35.794 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:35.794 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:35.794 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:32:35.795 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:35.795 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:35.795 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:35.795 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:32:35.796 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:35.796 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:32:35.797 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T07:32:35.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:35.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:35.799 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:35.842 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:35.843 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:36.045 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:32:36.046 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:32:36.046 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:32:36.046 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:32:36.047 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:32:36.047 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:36.047 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:36.047 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:36.048 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:32:36.048 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:36.048 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:32:36.048 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:36.048 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:36.049 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:32:36.049 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:36.049 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:32:36.049 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:36.049 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:32:36.050 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:36.050 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:36.050 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:36.050 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:36.051 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:32:36.051 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:36.051 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:36.051 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567955.431873_48.1", 2021-09-02T07:32:36.052 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:36.052 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:36.052 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:36.052 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:35.431873Z", 2021-09-02T07:32:36.052 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:36.053 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:36.053 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:36.053 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:36.053 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:32:36.054 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:36.054 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:36.054 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:36.054 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:36.055 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:36.055 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:36.055 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:36.055 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:36.055 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:36.056 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:36.056 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:36.056 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:32:36.056 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:36.057 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:36.057 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:36.057 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:36.057 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:36.058 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:36.058 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:36.058 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:36.058 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:36.058 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:36.059 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:32:36.059 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:36.059 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:32:36.059 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:32:36.060 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:32:36.060 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:36.060 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:32:36.060 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630567955.501542_59.1 target=1_1630567955.431873_48.1 2021-09-02T07:32:41.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:41.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:41.063 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:41.111 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:41.112 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:41.183 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:32:41.184 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:32:41.184 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:32:41.184 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:32:41.185 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:32:41.185 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:41.185 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:41.186 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:41.186 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:32:41.187 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:41.187 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:32:41.187 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:41.188 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:41.188 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:32:41.189 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:41.189 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:32:41.189 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:41.190 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:32:41.190 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:41.191 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:41.191 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:41.192 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:41.192 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:32:41.193 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:41.193 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:41.193 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567955.501542_59.1", 2021-09-02T07:32:41.194 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:41.194 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:41.194 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:41.195 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:35.501542Z", 2021-09-02T07:32:41.195 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:41.196 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:41.196 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:41.196 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:41.197 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:32:41.197 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:41.197 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:41.197 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:41.197 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:41.198 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:41.198 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:41.198 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:41.198 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:41.199 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:41.199 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:41.199 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:41.199 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:32:41.199 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:41.200 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:41.200 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:41.200 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:41.200 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:41.201 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:41.201 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:41.201 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:41.201 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:41.202 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:41.202 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:32:41.202 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:41.202 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:32:41.203 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:32:41.206 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:32:41.207 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:41.207 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:32:41.208 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T07:32:41.209 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T07:32:41.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:41.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:41.211 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:41.260 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:41.260 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:41.335 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:32:41.335 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:32:41.336 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:32:41.336 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:32:41.336 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:32:41.336 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:41.337 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:41.337 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:41.337 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:32:41.337 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:41.338 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:32:41.338 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:41.338 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:41.339 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:32:41.339 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:41.339 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:32:41.339 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:41.339 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:32:41.340 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:41.340 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:41.340 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:41.340 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:41.341 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:32:41.341 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:41.341 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:41.341 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567955.501542_59.1", 2021-09-02T07:32:41.342 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:41.342 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:41.342 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:41.342 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:35.501542Z", 2021-09-02T07:32:41.343 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:41.343 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:41.343 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:41.344 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:41.344 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:32:41.345 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:41.345 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:41.346 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:41.346 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:41.346 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:41.347 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:41.347 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:32:41.347 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:41.348 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:41.348 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:41.348 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:32:41.349 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:32:41.349 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:32:41.349 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:32:41.350 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:32:41.350 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:32:41.350 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:32:41.350 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:32:41.351 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:32:41.351 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:32:41.351 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:41.351 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:41.352 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:32:41.352 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:32:41.352 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:32:41.352 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:32:41.352 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:32:41.353 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:32:41.353 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:32:41.354 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T07:32:41.355 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T07:32:41.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:41.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:32:41.356 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:32:41.401 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:41.401 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:41.471 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:32:41.472 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:32:41.472 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:32:41.472 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:32:41.473 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:32:41.473 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:32:41.473 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:41.473 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:41.474 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:32:41.474 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:41.474 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:32:41.474 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:41.474 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:41.475 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:32:41.475 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:41.475 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:32:41.475 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:41.476 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:32:41.476 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:41.476 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:41.476 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:41.476 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:41.477 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:32:41.477 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:41.477 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:41.477 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567955.501542_59.1", 2021-09-02T07:32:41.478 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:41.478 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:41.478 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:41.478 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:35.501542Z", 2021-09-02T07:32:41.478 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:41.479 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:41.479 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:41.479 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:41.479 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:32:41.480 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:41.480 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:41.480 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:41.480 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:41.480 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:41.481 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:41.481 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:41.481 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:41.481 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:41.482 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:41.482 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:32:41.482 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:32:41.482 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:32:41.482 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:32:41.483 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:32:41.483 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:32:41.483 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:32:41.483 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:32:41.484 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:32:41.484 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:32:41.484 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:41.484 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:41.484 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:32:41.485 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:32:41.485 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:32:41.485 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:32:41.485 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:32:41.486 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:32:41.486 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:32:41.487 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-02T07:32:41.487 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:41.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:41.489 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:32:41.536 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:41.537 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:41.559 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:41.561+0000 7f19df950380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:32:41.559 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:41.561+0000 7f19df950380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:32:41.560 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:41.561+0000 7f19df950380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:32:41.560 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:41.561+0000 7f19df950380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:32:41.598 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:41.600+0000 7f19df950380 1 bucket sync caught up with source: 2021-09-02T07:32:41.598 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2021-09-02T07:32:41.599 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000001.6.5, , , , , , , , , ] 2021-09-02T07:32:41.599 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:41.600+0000 7f19df950380 0 bucket checkpoint complete 2021-09-02T07:32:41.613 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-13 zones={test-zone2, test-zone1} 2021-09-02T07:32:41.618 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:32:41.663 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:41.664 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:32:41.667 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:41.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:41.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:41.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:41.962 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-13 zones={test-zone2, test-zone1} 2021-09-02T07:32:41.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:41.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:41.964 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:32:42.014 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:42.014 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:42.036 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:42.038+0000 7f449537b380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:32:42.036 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:42.038+0000 7f449537b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:32:42.036 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:42.038+0000 7f449537b380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:32:42.037 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:42.038+0000 7f449537b380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:32:42.073 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:42.075+0000 7f449537b380 1 bucket sync caught up with source: 2021-09-02T07:32:42.073 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2021-09-02T07:32:42.074 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000001.6.5, , , , , , , , , ] 2021-09-02T07:32:42.074 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:42.075+0000 7f449537b380 0 bucket checkpoint complete 2021-09-02T07:32:42.089 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-13 zones={test-zone3, test-zone1} 2021-09-02T07:32:42.095 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:32:42.139 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:42.139 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:32:42.143 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:42.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:42.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:42.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:42.446 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-13 zones={test-zone3, test-zone1} 2021-09-02T07:32:42.447 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:42.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:42.448 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:32:42.502 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:42.503 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:42.527 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:42.529+0000 7f530082c380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:32:42.528 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:42.529+0000 7f530082c380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:32:42.528 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:42.529+0000 7f530082c380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:32:42.528 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:42.529+0000 7f530082c380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:32:42.569 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:42.572+0000 7f530082c380 1 bucket sync caught up with source: 2021-09-02T07:32:42.570 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2021-09-02T07:32:42.570 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000001.6.5, , , , , , , , , ] 2021-09-02T07:32:42.570 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:42.572+0000 7f530082c380 0 bucket checkpoint complete 2021-09-02T07:32:42.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:42.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:42.585 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:32:42.642 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:42.642 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:42.667 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:42.668+0000 7fbc10a74380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:32:42.667 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:42.668+0000 7fbc10a74380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:32:42.667 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:42.668+0000 7fbc10a74380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:32:42.667 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:42.668+0000 7fbc10a74380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:32:42.707 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:42.708+0000 7fbc10a74380 1 bucket sync caught up with source: 2021-09-02T07:32:42.707 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.164.5, , , , , , , , , ] 2021-09-02T07:32:42.707 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000001.164.5, , , , , , , , , ] 2021-09-02T07:32:42.708 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:42.708+0000 7fbc10a74380 0 bucket checkpoint complete 2021-09-02T07:32:42.724 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-14 zones={test-zone1, test-zone2} 2021-09-02T07:32:42.730 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:32:42.775 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:42.776 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:32:42.779 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:42.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:42.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:42.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:43.074 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-14 zones={test-zone1, test-zone2} 2021-09-02T07:32:43.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:43.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:43.076 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:32:43.126 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:43.126 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:43.147 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:43.149+0000 7f02b1ce1380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:32:43.148 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:43.149+0000 7f02b1ce1380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:32:43.148 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:43.149+0000 7f02b1ce1380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:32:43.148 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:43.149+0000 7f02b1ce1380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:32:43.184 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:43.186+0000 7f02b1ce1380 1 bucket sync caught up with source: 2021-09-02T07:32:43.184 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.164.5, , , , , , , , , ] 2021-09-02T07:32:43.185 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000001.164.5, , , , , , , , , ] 2021-09-02T07:32:43.185 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:43.186+0000 7f02b1ce1380 0 bucket checkpoint complete 2021-09-02T07:32:43.200 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-14 zones={test-zone3, test-zone2} 2021-09-02T07:32:43.206 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:32:43.251 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:43.252 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:32:43.255 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:43.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:43.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:43.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:43.563 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-14 zones={test-zone3, test-zone2} 2021-09-02T07:32:43.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:43.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:43.564 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:32:43.614 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:43.614 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:43.636 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:43.637+0000 7fdcc698d380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:32:43.636 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:43.637+0000 7fdcc698d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:32:43.636 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:43.637+0000 7fdcc698d380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:32:43.637 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:43.637+0000 7fdcc698d380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:32:43.670 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:43.672+0000 7fdcc698d380 1 bucket sync caught up with source: 2021-09-02T07:32:43.670 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.112.5, , , , , , , , , ] 2021-09-02T07:32:43.671 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000001.112.5, , , , , , , , , ] 2021-09-02T07:32:43.671 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:43.672+0000 7fdcc698d380 0 bucket checkpoint complete 2021-09-02T07:32:43.685 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-15 zones={test-zone1, test-zone3} 2021-09-02T07:32:43.691 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:32:43.735 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:43.736 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:32:43.739 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:43.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:43.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:43.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:44.038 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-15 zones={test-zone1, test-zone3} 2021-09-02T07:32:44.039 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:44.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:44.040 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:32:44.096 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:44.097 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:44.121 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:44.124+0000 7f1a4c168380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:32:44.122 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:44.124+0000 7f1a4c168380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:32:44.122 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:44.124+0000 7f1a4c168380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:32:44.122 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:44.124+0000 7f1a4c168380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:32:44.165 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:44.167+0000 7f1a4c168380 1 bucket sync caught up with source: 2021-09-02T07:32:44.165 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000001.112.5, , , , , , , , , ] 2021-09-02T07:32:44.165 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000001.112.5, , , , , , , , , ] 2021-09-02T07:32:44.166 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:44.167+0000 7f1a4c168380 0 bucket checkpoint complete 2021-09-02T07:32:44.183 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-15 zones={test-zone2, test-zone3} 2021-09-02T07:32:44.189 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:32:44.235 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:44.236 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:32:44.239 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-02T07:32:44.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:44.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:44.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-02T07:32:44.538 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-15 zones={test-zone2, test-zone3} 2021-09-02T07:32:44.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:44.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:44.548 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:32:44.597 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:44.598 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:44.619 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:44.621+0000 7fbda49d4380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:32:44.620 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:44.621+0000 7fbda49d4380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:32:44.620 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:44.621+0000 7fbda49d4380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:32:44.620 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:44.621+0000 7fbda49d4380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:32:44.656 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:44.658+0000 7fbda49d4380 1 waiting for incremental sync to catch up: 2021-09-02T07:32:44.657 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2021-09-02T07:32:44.657 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2021-09-02T07:32:49.658 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:49.660+0000 7fbda49d4380 1 bucket sync caught up with source: 2021-09-02T07:32:49.658 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000002.8.5, , , , , , , , , ] 2021-09-02T07:32:49.659 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2021-09-02T07:32:49.659 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:49.660+0000 7fbda49d4380 0 bucket checkpoint complete 2021-09-02T07:32:49.676 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-13 zones={test-zone2, test-zone1} 2021-09-02T07:32:49.683 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:32:49.727 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:32:49.782 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-13 zones={test-zone2, test-zone1} 2021-09-02T07:32:49.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:49.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:49.784 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:32:49.840 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:49.840 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:49.865 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:49.866+0000 7fbd8432f380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:32:49.865 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:49.866+0000 7fbd8432f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:32:49.865 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:49.866+0000 7fbd8432f380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:32:49.866 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:49.866+0000 7fbd8432f380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:32:49.896 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:49.898+0000 7fbd8432f380 1 bucket sync caught up with source: 2021-09-02T07:32:49.897 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000002.8.5, , , , , , , , , ] 2021-09-02T07:32:49.897 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2021-09-02T07:32:49.897 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:49.898+0000 7fbd8432f380 0 bucket checkpoint complete 2021-09-02T07:32:49.914 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-13 zones={test-zone3, test-zone1} 2021-09-02T07:32:49.921 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:32:49.971 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:32:50.022 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-13 zones={test-zone3, test-zone1} 2021-09-02T07:32:50.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:50.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:50.024 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:32:50.079 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:50.080 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:50.104 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:50.106+0000 7f98ecf2a380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:32:50.105 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:50.106+0000 7f98ecf2a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:32:50.105 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:50.106+0000 7f98ecf2a380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:32:50.105 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:50.106+0000 7f98ecf2a380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:32:50.146 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:50.148+0000 7f98ecf2a380 1 bucket sync caught up with source: 2021-09-02T07:32:50.146 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000002.8.5, , , , , , , , , ] 2021-09-02T07:32:50.146 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000002.8.5, , , , , , , , , ] 2021-09-02T07:32:50.147 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:32:50.148+0000 7f98ecf2a380 0 bucket checkpoint complete 2021-09-02T07:32:50.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:50.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:50.172 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:32:50.222 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:50.223 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:50.243 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:50.245+0000 7f48030e3380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:32:50.244 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:50.245+0000 7f48030e3380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:32:50.244 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:50.245+0000 7f48030e3380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:32:50.245 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:50.245+0000 7f48030e3380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:32:50.274 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:50.276+0000 7f48030e3380 1 waiting for incremental sync to catch up: 2021-09-02T07:32:50.275 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.164.5, , , , , , , , , ] 2021-09-02T07:32:50.275 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000002.186.5, , , , , , , , , ] 2021-09-02T07:32:55.277 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:55.278+0000 7f48030e3380 1 bucket sync caught up with source: 2021-09-02T07:32:55.277 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000002.186.5, , , , , , , , , ] 2021-09-02T07:32:55.277 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000002.186.5, , , , , , , , , ] 2021-09-02T07:32:55.278 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:55.278+0000 7f48030e3380 0 bucket checkpoint complete 2021-09-02T07:32:55.296 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-14 zones={test-zone1, test-zone2} 2021-09-02T07:32:55.303 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:32:55.351 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:32:55.401 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-14 zones={test-zone1, test-zone2} 2021-09-02T07:32:55.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:55.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:55.403 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:32:55.457 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:55.457 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:55.481 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:55.483+0000 7fa998a47380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:32:55.482 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:55.483+0000 7fa998a47380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:32:55.482 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:55.483+0000 7fa998a47380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:32:55.482 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:55.483+0000 7fa998a47380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:32:55.523 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:55.525+0000 7fa998a47380 1 bucket sync caught up with source: 2021-09-02T07:32:55.523 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000002.186.5, , , , , , , , , ] 2021-09-02T07:32:55.524 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000002.186.5, , , , , , , , , ] 2021-09-02T07:32:55.524 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:55.525+0000 7fa998a47380 0 bucket checkpoint complete 2021-09-02T07:32:55.538 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-14 zones={test-zone3, test-zone2} 2021-09-02T07:32:55.544 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:32:55.591 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:32:55.646 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-14 zones={test-zone3, test-zone2} 2021-09-02T07:32:55.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:55.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:32:55.657 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:32:55.707 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:32:55.707 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:32:55.729 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:55.731+0000 7f5b99100380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:32:55.729 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:55.731+0000 7f5b99100380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:32:55.730 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:55.731+0000 7f5b99100380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:32:55.730 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:55.731+0000 7f5b99100380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:32:55.759 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:32:55.761+0000 7f5b99100380 1 waiting for incremental sync to catch up: 2021-09-02T07:32:55.759 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000001.112.5, , , , , , , , , ] 2021-09-02T07:32:55.760 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000002.224.5, , , , , , , , , ] 2021-09-02T07:33:00.761 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:00.762+0000 7f5b99100380 1 bucket sync caught up with source: 2021-09-02T07:33:00.761 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000002.224.5, , , , , , , , , ] 2021-09-02T07:33:00.761 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000002.224.5, , , , , , , , , ] 2021-09-02T07:33:00.762 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:00.762+0000 7f5b99100380 0 bucket checkpoint complete 2021-09-02T07:33:00.779 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-15 zones={test-zone1, test-zone3} 2021-09-02T07:33:00.786 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:00.831 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:00.882 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-15 zones={test-zone1, test-zone3} 2021-09-02T07:33:00.883 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:00.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:00.884 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:00.941 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:00.941 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:00.965 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:00.967+0000 7fb3e2eb6380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:00.966 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:00.967+0000 7fb3e2eb6380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:00.966 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:00.967+0000 7fb3e2eb6380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:33:00.966 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:00.967+0000 7fb3e2eb6380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:01.011 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:01.012+0000 7fb3e2eb6380 1 bucket sync caught up with source: 2021-09-02T07:33:01.011 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000002.224.5, , , , , , , , , ] 2021-09-02T07:33:01.011 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000002.224.5, , , , , , , , , ] 2021-09-02T07:33:01.011 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:01.012+0000 7fb3e2eb6380 0 bucket checkpoint complete 2021-09-02T07:33:01.027 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-15 zones={test-zone2, test-zone3} 2021-09-02T07:33:01.033 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:01.079 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:01.134 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-15 zones={test-zone2, test-zone3} 2021-09-02T07:33:01.135 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2021-09-02T07:33:01.136 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-16 2021-09-02T07:33:01.157 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=efybpk-17 2021-09-02T07:33:01.194 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=efybpk-18 2021-09-02T07:33:01.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:33:01.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:33:01.261 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:33:01.303 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:01.303 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:01.362 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T07:33:01.379 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:33:01.379 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:33:01.380 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:33:01.380 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:32:10.734248Z" 2021-09-02T07:33:01.380 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:33:01.380 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:33:01.381 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567981.257754_82.1", 2021-09-02T07:33:01.381 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:33:01.257754Z" 2021-09-02T07:33:01.381 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:33:01.381 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:33:01.382 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:33:01.382 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:33:01.382 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:33:01.382 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:33:01.383 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:33:01.383 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:33:01.383 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:33:01.383 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:33:01.384 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T07:33:01.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:33:01.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:33:01.386 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:33:01.432 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:01.432 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:01.498 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:33:01.498 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:33:01.499 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:33:01.499 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:33:01.499 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:33:01.499 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:33:01.500 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:01.500 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:01.500 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:33:01.500 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:01.500 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:33:01.501 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:01.501 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:01.501 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:33:01.501 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:01.501 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:33:01.502 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:01.502 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:33:01.502 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:01.502 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:01.503 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:01.503 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:01.503 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:33:01.503 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:01.504 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:01.504 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567981.147716_65.1", 2021-09-02T07:33:01.504 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:01.504 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:33:01.504 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:01.505 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:33:01.147716Z", 2021-09-02T07:33:01.505 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:01.505 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:01.505 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:01.505 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:01.506 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:33:01.506 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:01.506 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:01.506 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:33:01.507 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:01.507 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:33:01.507 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:01.507 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:33:01.507 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:01.508 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:01.508 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:01.508 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:01.508 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:33:01.509 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:01.509 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:01.509 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:33:01.509 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:01.509 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:33:01.510 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:01.510 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:33:01.510 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:01.510 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:01.510 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:01.511 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:33:01.511 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:01.511 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:33:01.511 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:33:01.512 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:33:01.512 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:01.512 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:33:01.513 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630567981.257754_82.1 target=1_1630567981.147716_65.1 2021-09-02T07:33:06.514 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:33:06.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:33:06.516 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:33:06.563 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:06.564 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:06.629 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:33:06.629 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:33:06.629 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:33:06.630 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:33:06.630 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:33:06.630 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:33:06.630 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:06.631 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:06.631 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:33:06.631 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:06.631 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:33:06.631 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:06.632 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:06.632 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:33:06.632 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:06.632 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:33:06.633 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:06.633 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:33:06.633 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:06.633 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:06.634 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:06.634 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:06.634 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:33:06.634 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:06.635 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:06.635 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567981.257754_82.1", 2021-09-02T07:33:06.635 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:06.635 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:33:06.636 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:06.636 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:33:01.257754Z", 2021-09-02T07:33:06.636 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:06.636 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:06.637 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:06.637 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:06.637 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:33:06.637 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:06.638 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:06.638 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:33:06.638 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:06.638 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:33:06.639 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:06.639 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:33:06.639 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:06.640 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:06.640 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:06.640 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:06.640 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:33:06.640 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:06.641 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:06.641 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:33:06.641 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:06.641 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:33:06.642 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:06.642 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:33:06.642 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:06.642 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:06.643 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:06.643 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:33:06.643 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:06.643 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:33:06.644 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:33:06.644 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:33:06.644 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:06.644 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:33:06.646 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T07:33:06.646 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T07:33:06.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:33:06.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:33:06.648 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:33:06.698 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:06.698 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:06.777 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:33:06.777 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:33:06.777 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:33:06.778 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:33:06.778 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:33:06.778 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:33:06.778 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:33:06.779 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:33:06.779 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:33:06.779 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:33:06.780 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:33:06.780 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:33:06.780 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:33:06.780 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:33:06.781 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:33:06.781 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:33:06.781 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:33:06.781 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:33:06.782 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:33:06.782 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:33:06.782 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:33:06.782 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:33:06.783 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:33:06.783 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:33:06.783 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:33:06.783 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567981.257754_82.1", 2021-09-02T07:33:06.784 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:33:06.784 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:33:06.784 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:33:06.784 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:33:01.257754Z", 2021-09-02T07:33:06.784 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:33:06.785 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:33:06.785 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:33:06.785 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:33:06.785 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:33:06.786 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:33:06.786 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:33:06.786 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:33:06.786 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:33:06.787 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:33:06.787 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:33:06.787 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:33:06.787 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:33:06.788 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:33:06.788 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:33:06.788 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:33:06.788 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:33:06.789 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:33:06.789 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:33:06.790 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:33:06.790 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:33:06.790 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:33:06.790 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:33:06.790 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:33:06.791 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:33:06.791 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:33:06.791 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:33:06.791 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:33:06.792 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:33:06.792 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:33:06.792 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:33:06.792 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:33:06.792 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:33:06.793 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:33:06.794 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T07:33:06.795 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T07:33:06.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:33:06.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:33:06.797 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:33:06.841 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:06.842 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:06.904 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:33:06.905 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:33:06.905 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:33:06.905 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:33:06.905 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:33:06.906 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:33:06.906 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:06.906 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:06.906 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:33:06.907 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:06.907 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:33:06.907 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:06.907 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:06.908 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:33:06.908 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:06.908 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:33:06.908 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:06.908 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:33:06.909 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:06.909 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:06.909 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:06.909 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:06.910 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:33:06.910 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:06.910 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:06.910 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567981.257754_82.1", 2021-09-02T07:33:06.911 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:06.911 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:33:06.911 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:06.911 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:33:01.257754Z", 2021-09-02T07:33:06.911 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:06.912 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:06.912 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:06.912 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:06.912 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:33:06.913 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:06.913 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:06.913 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:33:06.913 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:06.913 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:33:06.914 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:06.914 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:33:06.914 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:06.914 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:06.915 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:06.915 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:06.915 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:33:06.915 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:06.916 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:06.916 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:33:06.916 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:06.916 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:33:06.916 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:06.917 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:33:06.917 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:06.917 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:06.917 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:06.918 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:33:06.918 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:06.918 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:33:06.918 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:33:06.918 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:33:06.919 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:06.919 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:33:06.920 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-02T07:33:06.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:06.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:06.968 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:07.017 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:07.018 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:07.039 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:07.040+0000 7fecf1538380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:07.039 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:07.040+0000 7fecf1538380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:07.040 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:07.040+0000 7fecf1538380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:33:07.040 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:07.040+0000 7fecf1538380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:07.081 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:07.082+0000 7fecf1538380 1 waiting for incremental sync to catch up: 2021-09-02T07:33:07.081 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , , , ] 2021-09-02T07:33:07.081 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , 00000000002.7.10, , , , , ] 2021-09-02T07:33:12.083 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:12.084+0000 7fecf1538380 1 bucket sync caught up with source: 2021-09-02T07:33:12.083 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , 00000000006.15.9, , , , , ] 2021-09-02T07:33:12.083 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , 00000000002.7.10, , , , , ] 2021-09-02T07:33:12.084 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:12.084+0000 7fecf1538380 0 bucket checkpoint complete 2021-09-02T07:33:12.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:12.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:12.103 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:12.158 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:12.158 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:12.183 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:12.186+0000 7f8f3d98a380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:12.184 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:12.186+0000 7f8f3d98a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:12.184 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:12.186+0000 7f8f3d98a380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:33:12.184 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:12.186+0000 7f8f3d98a380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:12.224 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:12.227+0000 7f8f3d98a380 1 bucket sync caught up with source: 2021-09-02T07:33:12.225 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000006.15.9, , , , , ] 2021-09-02T07:33:12.225 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000006.15.9, , , , , ] 2021-09-02T07:33:12.225 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:12.227+0000 7f8f3d98a380 0 bucket checkpoint complete 2021-09-02T07:33:12.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:12.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:12.241 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:12.296 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:12.296 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:12.321 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:12.323+0000 7fe213262380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:12.322 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:12.323+0000 7fe213262380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:12.322 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:12.323+0000 7fe213262380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:33:12.322 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:12.323+0000 7fe213262380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:12.355 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:12.356+0000 7fe213262380 1 bucket sync caught up with source: 2021-09-02T07:33:12.355 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , 00000000006.15.9, , , , , ] 2021-09-02T07:33:12.356 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , 00000000006.15.9, , , , , ] 2021-09-02T07:33:12.356 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:12.356+0000 7fe213262380 0 bucket checkpoint complete 2021-09-02T07:33:12.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:12.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:12.421 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:12.470 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:12.471 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:12.494 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:12.496+0000 7f3cc20f4380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:12.494 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:12.496+0000 7f3cc20f4380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:12.494 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:12.496+0000 7f3cc20f4380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:33:12.495 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:12.496+0000 7f3cc20f4380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:12.526 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:12.529+0000 7f3cc20f4380 1 waiting for incremental sync to catch up: 2021-09-02T07:33:12.527 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , , , ] 2021-09-02T07:33:12.527 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000002.115.10, , , , , ] 2021-09-02T07:33:17.528 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:17.530+0000 7f3cc20f4380 1 bucket sync caught up with source: 2021-09-02T07:33:17.529 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000006.123.9, , , , , ] 2021-09-02T07:33:17.529 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000002.115.10, , , , , ] 2021-09-02T07:33:17.529 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:17.530+0000 7f3cc20f4380 0 bucket checkpoint complete 2021-09-02T07:33:17.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:17.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:17.549 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:17.603 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:17.603 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:17.626 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:17.628+0000 7f3e375cc380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:17.627 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:17.628+0000 7f3e375cc380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:17.627 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:17.628+0000 7f3e375cc380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:33:17.627 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:17.628+0000 7f3e375cc380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:17.660 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:17.661+0000 7f3e375cc380 1 bucket sync caught up with source: 2021-09-02T07:33:17.660 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000006.123.9, , , , , ] 2021-09-02T07:33:17.661 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000006.123.9, , , , , ] 2021-09-02T07:33:17.661 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:17.661+0000 7f3e375cc380 0 bucket checkpoint complete 2021-09-02T07:33:17.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:17.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:17.725 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:17.779 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:17.779 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:17.802 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:17.803+0000 7fc060296380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:17.802 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:17.803+0000 7fc060296380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:17.802 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:17.803+0000 7fc060296380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:33:17.803 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:17.803+0000 7fc060296380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:17.843 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:17.844+0000 7fc060296380 1 waiting for incremental sync to catch up: 2021-09-02T07:33:17.843 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , , , ] 2021-09-02T07:33:17.843 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000002.253.10, , , , , ] 2021-09-02T07:33:22.844 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:22.846+0000 7fc060296380 1 bucket sync caught up with source: 2021-09-02T07:33:22.845 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000006.261.9, , , , , ] 2021-09-02T07:33:22.845 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000002.253.10, , , , , ] 2021-09-02T07:33:22.845 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:22.846+0000 7fc060296380 0 bucket checkpoint complete 2021-09-02T07:33:22.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:22.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:22.867 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:22.922 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:22.923 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:22.948 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:22.950+0000 7f78a3f21380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:22.948 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:22.950+0000 7f78a3f21380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:22.948 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:22.950+0000 7f78a3f21380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:33:22.948 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:22.950+0000 7f78a3f21380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:22.992 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:22.994+0000 7f78a3f21380 1 bucket sync caught up with source: 2021-09-02T07:33:22.992 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , 00000000006.261.9, , , , , ] 2021-09-02T07:33:22.992 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , 00000000006.261.9, , , , , ] 2021-09-02T07:33:22.993 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:22.994+0000 7f78a3f21380 0 bucket checkpoint complete 2021-09-02T07:33:24.535 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:24.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:24.537 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:24.588 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:24.588 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:24.612 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:24.614+0000 7f5b21d56380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:24.613 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:24.614+0000 7f5b21d56380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:24.613 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:24.614+0000 7f5b21d56380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:33:24.613 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:24.614+0000 7f5b21d56380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:24.655 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:24.657+0000 7f5b21d56380 1 bucket sync caught up with source: 2021-09-02T07:33:24.656 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000010.24.5, , 00000000003.9.9, , 00000000009.21.9, , , , , ] 2021-09-02T07:33:24.656 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000010.24.5, , 00000000003.9.9, , 00000000009.21.9, , , , , ] 2021-09-02T07:33:24.656 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:24.657+0000 7f5b21d56380 0 bucket checkpoint complete 2021-09-02T07:33:24.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:24.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:24.675 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:24.727 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:24.727 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:24.750 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:24.751+0000 7f7f784cb380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:24.750 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:24.751+0000 7f7f784cb380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:24.750 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:24.751+0000 7f7f784cb380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:33:24.751 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:24.751+0000 7f7f784cb380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:24.788 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:24.789+0000 7f7f784cb380 1 bucket sync caught up with source: 2021-09-02T07:33:24.788 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000010.24.5, , 00000000003.9.9, , 00000000009.21.9, , , , , ] 2021-09-02T07:33:24.788 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000010.24.5, , 00000000003.9.9, , 00000000009.21.9, , , , , ] 2021-09-02T07:33:24.789 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:24.789+0000 7f7f784cb380 0 bucket checkpoint complete 2021-09-02T07:33:24.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:24.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:24.804 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:24.853 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:24.854 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:24.874 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:24.877+0000 7f4ce8afd380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:24.875 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:24.877+0000 7f4ce8afd380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:24.875 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:24.877+0000 7f4ce8afd380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:33:24.875 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:24.877+0000 7f4ce8afd380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:24.905 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:24.908+0000 7f4ce8afd380 1 bucket sync caught up with source: 2021-09-02T07:33:24.906 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000010.24.5, , 00000000003.9.9, , 00000000009.21.9, , , , , ] 2021-09-02T07:33:24.906 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000010.24.5, , 00000000003.9.9, , 00000000009.21.9, , , , , ] 2021-09-02T07:33:24.906 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:24.908+0000 7f4ce8afd380 0 bucket checkpoint complete 2021-09-02T07:33:24.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:24.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:24.923 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:24.972 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:24.972 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:24.994 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:24.996+0000 7fd32ec82380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:24.994 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:24.996+0000 7fd32ec82380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:24.994 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:24.996+0000 7fd32ec82380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:33:24.995 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:24.996+0000 7fd32ec82380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:25.025 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:25.027+0000 7fd32ec82380 1 bucket sync caught up with source: 2021-09-02T07:33:25.026 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.139.9, , 00000000010.189.5, , 00000000007.164.9, , , , , ] 2021-09-02T07:33:25.026 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.139.9, , 00000000010.189.5, , 00000000007.164.9, , , , , ] 2021-09-02T07:33:25.026 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:25.027+0000 7fd32ec82380 0 bucket checkpoint complete 2021-09-02T07:33:25.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:25.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:25.041 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:25.090 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:25.090 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:25.112 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:25.114+0000 7f540d589380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:25.113 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:25.114+0000 7f540d589380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:25.113 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:25.114+0000 7f540d589380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:33:25.113 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:25.114+0000 7f540d589380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:25.143 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:25.145+0000 7f540d589380 1 bucket sync caught up with source: 2021-09-02T07:33:25.144 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.139.9, , 00000000010.189.5, , 00000000007.164.9, , , , , ] 2021-09-02T07:33:25.144 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.139.9, , 00000000010.189.5, , 00000000007.164.9, , , , , ] 2021-09-02T07:33:25.144 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:25.145+0000 7f540d589380 0 bucket checkpoint complete 2021-09-02T07:33:25.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:25.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:25.160 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:25.207 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:25.208 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:25.230 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:25.232+0000 7f988d9d2380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:25.230 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:25.232+0000 7f988d9d2380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:25.231 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:25.232+0000 7f988d9d2380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:33:25.231 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:25.232+0000 7f988d9d2380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:25.261 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:25.263+0000 7f988d9d2380 1 bucket sync caught up with source: 2021-09-02T07:33:25.261 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000005.152.5, , 00000000003.136.9, , 00000000014.148.5, , , , , ] 2021-09-02T07:33:25.262 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000005.152.5, , 00000000003.136.9, , 00000000014.148.5, , , , , ] 2021-09-02T07:33:25.262 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:25.263+0000 7f988d9d2380 0 bucket checkpoint complete 2021-09-02T07:33:25.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:25.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:25.281 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:25.330 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:25.330 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:25.353 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:25.356+0000 7f412d042380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:25.354 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:25.356+0000 7f412d042380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:25.354 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:25.356+0000 7f412d042380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:33:25.355 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:25.356+0000 7f412d042380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:25.385 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:25.387+0000 7f412d042380 1 bucket sync caught up with source: 2021-09-02T07:33:25.385 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000005.152.5, , 00000000003.136.9, , 00000000014.148.5, , , , , ] 2021-09-02T07:33:25.385 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000005.152.5, , 00000000003.136.9, , 00000000014.148.5, , , , , ] 2021-09-02T07:33:25.385 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:25.387+0000 7f412d042380 0 bucket checkpoint complete 2021-09-02T07:33:25.401 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-16 zones={test-zone2, test-zone1} 2021-09-02T07:33:25.407 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:25.411 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:25.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:25.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:25.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:25.413 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:25.416 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:25.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:25.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:25.418 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:25.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:25.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:25.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:25.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:25.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:25.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:25.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:25.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:25.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:26.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:26.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:26.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:26.318 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-16 zones={test-zone2, test-zone1} 2021-09-02T07:33:26.319 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-16 zones={test-zone3, test-zone1} 2021-09-02T07:33:26.325 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:26.328 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:26.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:26.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:26.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:26.332 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:26.335 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:26.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:26.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:26.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:26.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:26.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:26.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:26.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:26.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:26.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:26.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:26.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:26.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:27.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:27.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:27.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:27.271 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-16 zones={test-zone3, test-zone1} 2021-09-02T07:33:27.272 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-16 zones={test-zone3, test-zone2} 2021-09-02T07:33:27.277 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:27.281 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:27.281 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:27.282 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:27.282 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:27.283 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:27.287 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:27.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:27.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:27.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:27.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:27.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:27.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:27.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:27.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:27.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:27.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:27.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:27.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:27.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:28.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:28.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:28.222 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-16 zones={test-zone3, test-zone2} 2021-09-02T07:33:28.223 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:28.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:28.224 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:28.278 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:28.279 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:28.303 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:28.305+0000 7f0b57390380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:28.304 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:28.305+0000 7f0b57390380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:28.304 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:28.305+0000 7f0b57390380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:33:28.304 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:28.305+0000 7f0b57390380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:28.337 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:28.339+0000 7f0b57390380 1 bucket sync caught up with source: 2021-09-02T07:33:28.337 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000010.28.5, , 00000000003.11.9, , 00000000011.24.5, , , , , ] 2021-09-02T07:33:28.338 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000010.28.5, , 00000000003.11.9, , 00000000011.24.5, , , , , ] 2021-09-02T07:33:28.338 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:28.339+0000 7f0b57390380 0 bucket checkpoint complete 2021-09-02T07:33:28.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:28.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:28.356 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:28.405 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:28.405 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:28.427 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.429+0000 7fb2128b9380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:28.427 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.429+0000 7fb2128b9380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:28.427 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.429+0000 7fb2128b9380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:33:28.427 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.429+0000 7fb2128b9380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:28.459 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.461+0000 7fb2128b9380 1 bucket sync caught up with source: 2021-09-02T07:33:28.459 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000010.28.5, , 00000000003.11.9, , 00000000011.24.5, , , , , ] 2021-09-02T07:33:28.459 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000010.28.5, , 00000000003.11.9, , 00000000011.24.5, , , , , ] 2021-09-02T07:33:28.460 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.461+0000 7fb2128b9380 0 bucket checkpoint complete 2021-09-02T07:33:28.474 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:28.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:28.475 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:28.529 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:28.529 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:28.551 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:28.553+0000 7f4b4baf6380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:28.551 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:28.553+0000 7f4b4baf6380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:28.551 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:28.553+0000 7f4b4baf6380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:33:28.552 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:28.553+0000 7f4b4baf6380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:28.592 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:28.594+0000 7f4b4baf6380 1 bucket sync caught up with source: 2021-09-02T07:33:28.592 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000010.28.5, , 00000000003.11.9, , 00000000011.24.5, , , , , ] 2021-09-02T07:33:28.592 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000010.28.5, , 00000000003.11.9, , 00000000011.24.5, , , , , ] 2021-09-02T07:33:28.592 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:28.594+0000 7f4b4baf6380 0 bucket checkpoint complete 2021-09-02T07:33:28.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:28.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:28.614 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:28.663 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:28.663 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:28.685 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.687+0000 7f2146760380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:28.685 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.687+0000 7f2146760380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:28.686 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.687+0000 7f2146760380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:33:28.686 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.687+0000 7f2146760380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:28.715 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.717+0000 7f2146760380 1 bucket sync caught up with source: 2021-09-02T07:33:28.716 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.290.9, , 00000000010.199.5, , 00000000012.136.5, , , , , ] 2021-09-02T07:33:28.716 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.290.9, , 00000000010.199.5, , 00000000012.136.5, , , , , ] 2021-09-02T07:33:28.716 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.717+0000 7f2146760380 0 bucket checkpoint complete 2021-09-02T07:33:28.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:28.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:28.732 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:28.780 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:28.780 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:28.806 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.808+0000 7f636ec57380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:28.807 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.808+0000 7f636ec57380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:28.807 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.808+0000 7f636ec57380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:33:28.807 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.808+0000 7f636ec57380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:28.840 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.842+0000 7f636ec57380 1 bucket sync caught up with source: 2021-09-02T07:33:28.841 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.290.9, , 00000000010.199.5, , 00000000012.136.5, , , , , ] 2021-09-02T07:33:28.841 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.290.9, , 00000000010.199.5, , 00000000012.136.5, , , , , ] 2021-09-02T07:33:28.841 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.842+0000 7f636ec57380 0 bucket checkpoint complete 2021-09-02T07:33:28.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:28.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:28.859 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:28.909 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:28.909 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:28.933 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.935+0000 7f08541af380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:28.933 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.935+0000 7f08541af380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:28.934 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.935+0000 7f08541af380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:33:28.934 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.935+0000 7f08541af380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:28.973 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.974+0000 7f08541af380 1 bucket sync caught up with source: 2021-09-02T07:33:28.973 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.265.9, , 00000000005.163.5, , 00000000014.126.5, , , , , ] 2021-09-02T07:33:28.973 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.265.9, , 00000000005.163.5, , 00000000014.126.5, , , , , ] 2021-09-02T07:33:28.973 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:28.974+0000 7f08541af380 0 bucket checkpoint complete 2021-09-02T07:33:28.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:28.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:28.989 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:29.039 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:29.039 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:29.061 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:29.063+0000 7f0b029cc380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:29.061 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:29.063+0000 7f0b029cc380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:29.062 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:29.063+0000 7f0b029cc380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:33:29.062 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:29.063+0000 7f0b029cc380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:29.103 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:29.105+0000 7f0b029cc380 1 bucket sync caught up with source: 2021-09-02T07:33:29.103 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000003.265.9, , 00000000005.163.5, , 00000000014.126.5, , , , , ] 2021-09-02T07:33:29.103 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000003.265.9, , 00000000005.163.5, , 00000000014.126.5, , , , , ] 2021-09-02T07:33:29.104 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:29.105+0000 7f0b029cc380 0 bucket checkpoint complete 2021-09-02T07:33:29.119 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-17 zones={test-zone2, test-zone1} 2021-09-02T07:33:29.125 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:29.129 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:29.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:29.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:29.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:29.131 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:29.135 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:29.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:29.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:29.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:29.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:29.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:29.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:29.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:29.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:29.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:29.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:29.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:29.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:29.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:29.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:29.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:30.026 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-17 zones={test-zone2, test-zone1} 2021-09-02T07:33:30.026 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-17 zones={test-zone3, test-zone1} 2021-09-02T07:33:30.032 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:30.036 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:30.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:30.039 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:30.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:30.041 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:30.045 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:30.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:30.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:30.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:30.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:30.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:30.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:30.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:30.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:30.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:30.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:30.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:30.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:30.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:30.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:30.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:30.934 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-17 zones={test-zone3, test-zone1} 2021-09-02T07:33:30.935 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-17 zones={test-zone3, test-zone2} 2021-09-02T07:33:30.940 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:30.947 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:30.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:30.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:30.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:30.949 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:30.953 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:30.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:30.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:30.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:30.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:30.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:31.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:31.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:31.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:31.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:31.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:31.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:31.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:31.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:31.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:31.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:31.854 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-17 zones={test-zone3, test-zone2} 2021-09-02T07:33:31.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:31.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:31.856 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:31.911 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:31.912 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:31.937 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:31.939+0000 7f0cf12d6380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:31.937 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:31.939+0000 7f0cf12d6380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:31.937 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:31.939+0000 7f0cf12d6380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:33:31.938 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:31.939+0000 7f0cf12d6380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:31.979 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:31.981+0000 7f0cf12d6380 1 bucket sync caught up with source: 2021-09-02T07:33:31.979 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000010.22.5, , 00000000003.12.9, , 00000000007.27.9, , , , , ] 2021-09-02T07:33:31.979 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000010.22.5, , 00000000003.12.9, , 00000000007.27.9, , , , , ] 2021-09-02T07:33:31.980 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:31.981+0000 7f0cf12d6380 0 bucket checkpoint complete 2021-09-02T07:33:31.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:31.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:31.995 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:32.047 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:32.048 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:32.069 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.071+0000 7f7ca9c33380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:32.069 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.071+0000 7f7ca9c33380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:32.070 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.071+0000 7f7ca9c33380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:33:32.070 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.071+0000 7f7ca9c33380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:32.101 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.103+0000 7f7ca9c33380 1 bucket sync caught up with source: 2021-09-02T07:33:32.102 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000010.22.5, , 00000000003.12.9, , 00000000007.27.9, , , , , ] 2021-09-02T07:33:32.102 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000010.22.5, , 00000000003.12.9, , 00000000007.27.9, , , , , ] 2021-09-02T07:33:32.102 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.103+0000 7f7ca9c33380 0 bucket checkpoint complete 2021-09-02T07:33:32.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:32.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:32.120 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:32.169 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:32.169 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:32.191 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:32.193+0000 7fc510cc9380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:32.191 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:32.193+0000 7fc510cc9380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:32.192 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:32.193+0000 7fc510cc9380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:33:32.192 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:32.193+0000 7fc510cc9380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:32.224 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:32.226+0000 7fc510cc9380 1 bucket sync caught up with source: 2021-09-02T07:33:32.224 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000010.22.5, , 00000000003.12.9, , 00000000007.27.9, , , , , ] 2021-09-02T07:33:32.224 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000010.22.5, , 00000000003.12.9, , 00000000007.27.9, , , , , ] 2021-09-02T07:33:32.225 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:32.226+0000 7fc510cc9380 0 bucket checkpoint complete 2021-09-02T07:33:32.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:32.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:32.240 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:32.291 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:32.291 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:32.311 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.313+0000 7fa2f0cd4380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:32.312 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.313+0000 7fa2f0cd4380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:32.312 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.313+0000 7fa2f0cd4380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:33:32.312 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.313+0000 7fa2f0cd4380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:32.352 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.353+0000 7fa2f0cd4380 1 bucket sync caught up with source: 2021-09-02T07:33:32.352 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.114.9, , 00000000010.289.5, , 00000000007.279.9, , , , , ] 2021-09-02T07:33:32.352 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.114.9, , 00000000010.289.5, , 00000000007.279.9, , , , , ] 2021-09-02T07:33:32.352 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.353+0000 7fa2f0cd4380 0 bucket checkpoint complete 2021-09-02T07:33:32.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:32.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:32.368 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:32.417 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:32.418 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:32.441 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.443+0000 7f7bb99c4380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:32.442 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.443+0000 7f7bb99c4380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:32.442 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.443+0000 7f7bb99c4380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:33:32.442 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.443+0000 7f7bb99c4380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:32.482 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.484+0000 7f7bb99c4380 1 bucket sync caught up with source: 2021-09-02T07:33:32.482 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.114.9, , 00000000010.289.5, , 00000000007.279.9, , , , , ] 2021-09-02T07:33:32.482 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.114.9, , 00000000010.289.5, , 00000000007.279.9, , , , , ] 2021-09-02T07:33:32.483 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.484+0000 7f7bb99c4380 0 bucket checkpoint complete 2021-09-02T07:33:32.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:32.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:32.500 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:32.548 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:32.549 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:32.572 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.574+0000 7f9347c05380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:32.572 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.574+0000 7f9347c05380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:32.572 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.574+0000 7f9347c05380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:33:32.573 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.574+0000 7f9347c05380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:32.613 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.615+0000 7f9347c05380 1 bucket sync caught up with source: 2021-09-02T07:33:32.613 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000003.107.9, , 00000000003.227.9, , 00000000016.281.5, , , , , ] 2021-09-02T07:33:32.614 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000003.107.9, , 00000000003.227.9, , 00000000016.281.5, , , , , ] 2021-09-02T07:33:32.614 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:32.615+0000 7f9347c05380 0 bucket checkpoint complete 2021-09-02T07:33:32.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:32.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:32.630 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:32.682 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:32.683 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:32.705 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:32.707+0000 7f0b17c90380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:32.705 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:32.707+0000 7f0b17c90380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:32.705 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:32.707+0000 7f0b17c90380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:33:32.706 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:32.707+0000 7f0b17c90380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:32.752 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:32.754+0000 7f0b17c90380 1 bucket sync caught up with source: 2021-09-02T07:33:32.752 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000003.107.9, , 00000000003.227.9, , 00000000016.281.5, , , , , ] 2021-09-02T07:33:32.752 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000003.107.9, , 00000000003.227.9, , 00000000016.281.5, , , , , ] 2021-09-02T07:33:32.753 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:32.754+0000 7f0b17c90380 0 bucket checkpoint complete 2021-09-02T07:33:32.767 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-18 zones={test-zone2, test-zone1} 2021-09-02T07:33:32.774 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:32.777 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:32.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:32.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:32.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:32.780 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:32.783 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:32.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:32.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:32.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:32.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:32.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:32.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:32.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:32.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:33.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:33.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:33.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:33.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:33.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:33.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:33.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:33.654 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-18 zones={test-zone2, test-zone1} 2021-09-02T07:33:33.655 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-18 zones={test-zone3, test-zone1} 2021-09-02T07:33:33.660 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:33.664 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:33.664 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:33.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:33.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:33.666 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:33.669 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:33.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:33.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:33.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:33.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:33.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:33.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:33.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:33.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:33.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:34.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:34.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:34.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:34.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:34.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:34.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:34.554 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-18 zones={test-zone3, test-zone1} 2021-09-02T07:33:34.555 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-18 zones={test-zone3, test-zone2} 2021-09-02T07:33:34.561 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:34.564 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:34.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:34.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:34.566 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:34.566 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:34.570 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:34.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:34.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:34.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:34.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:34.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:34.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:34.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:34.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:34.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:34.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:35.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:35.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:35.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:35.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:35.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:35.511 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-18 zones={test-zone3, test-zone2} 2021-09-02T07:33:36.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:36.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:36.033 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:36.084 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:36.085 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:36.107 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:36.109+0000 7fa5aa518380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:36.107 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:36.109+0000 7fa5aa518380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:36.107 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:36.109+0000 7fa5aa518380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:33:36.107 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:36.109+0000 7fa5aa518380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:36.151 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:36.153+0000 7fa5aa518380 1 bucket sync caught up with source: 2021-09-02T07:33:36.151 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000011.29.5, , 00000000004.12.5, , 00000000010.24.5, , , , , ] 2021-09-02T07:33:36.151 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000011.29.5, , 00000000004.12.5, , 00000000010.24.5, , , , , ] 2021-09-02T07:33:36.151 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:36.153+0000 7fa5aa518380 0 bucket checkpoint complete 2021-09-02T07:33:36.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:36.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:36.171 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:36.225 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:36.226 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:36.247 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.249+0000 7fade6bd4380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:36.248 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.249+0000 7fade6bd4380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:36.248 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.249+0000 7fade6bd4380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:33:36.249 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.249+0000 7fade6bd4380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:36.284 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.286+0000 7fade6bd4380 1 bucket sync caught up with source: 2021-09-02T07:33:36.284 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000011.29.5, , 00000000004.12.5, , 00000000010.24.5, , , , , ] 2021-09-02T07:33:36.285 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000011.29.5, , 00000000004.12.5, , 00000000010.24.5, , , , , ] 2021-09-02T07:33:36.285 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.286+0000 7fade6bd4380 0 bucket checkpoint complete 2021-09-02T07:33:36.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:36.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:36.301 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:36.352 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:36.352 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:36.378 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:36.380+0000 7f6658601380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:36.378 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:36.380+0000 7f6658601380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:36.378 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:36.380+0000 7f6658601380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:33:36.379 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:36.380+0000 7f6658601380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:36.408 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:36.411+0000 7f6658601380 1 bucket sync caught up with source: 2021-09-02T07:33:36.409 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000011.29.5, , 00000000004.12.5, , 00000000010.24.5, , , , , ] 2021-09-02T07:33:36.409 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000011.29.5, , 00000000004.12.5, , 00000000010.24.5, , , , , ] 2021-09-02T07:33:36.409 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:36.411+0000 7f6658601380 0 bucket checkpoint complete 2021-09-02T07:33:36.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:36.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:36.425 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:36.474 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:36.475 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:36.496 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.498+0000 7f28f20e9380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:36.497 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.498+0000 7f28f20e9380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:36.497 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.498+0000 7f28f20e9380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:33:36.497 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.498+0000 7f28f20e9380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:36.526 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.528+0000 7f28f20e9380 1 bucket sync caught up with source: 2021-09-02T07:33:36.526 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000004.142.5, , 00000000011.196.5, , 00000000008.167.5, , , , , ] 2021-09-02T07:33:36.527 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000004.142.5, , 00000000011.196.5, , 00000000008.167.5, , , , , ] 2021-09-02T07:33:36.527 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.528+0000 7f28f20e9380 0 bucket checkpoint complete 2021-09-02T07:33:36.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:36.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:36.542 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:36.591 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:36.591 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:36.613 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.615+0000 7fd22332d380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:36.613 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.615+0000 7fd22332d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:36.614 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.615+0000 7fd22332d380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:33:36.614 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.615+0000 7fd22332d380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:36.643 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.645+0000 7fd22332d380 1 bucket sync caught up with source: 2021-09-02T07:33:36.643 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000004.142.5, , 00000000011.196.5, , 00000000008.167.5, , , , , ] 2021-09-02T07:33:36.644 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000004.142.5, , 00000000011.196.5, , 00000000008.167.5, , , , , ] 2021-09-02T07:33:36.644 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.645+0000 7fd22332d380 0 bucket checkpoint complete 2021-09-02T07:33:36.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:36.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:36.662 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:36.711 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:36.711 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:36.732 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.734+0000 7f42b023f380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:36.733 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.734+0000 7f42b023f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:36.733 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.734+0000 7f42b023f380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:33:36.733 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.734+0000 7f42b023f380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:36.773 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.775+0000 7f42b023f380 1 bucket sync caught up with source: 2021-09-02T07:33:36.773 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000006.156.5, , 00000000004.139.5, , 00000000015.151.5, , , , , ] 2021-09-02T07:33:36.774 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000006.156.5, , 00000000004.139.5, , 00000000015.151.5, , , , , ] 2021-09-02T07:33:36.774 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:36.775+0000 7f42b023f380 0 bucket checkpoint complete 2021-09-02T07:33:36.788 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:36.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:36.790 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:36.843 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:36.843 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:36.865 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:36.866+0000 7f99a486b380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:36.866 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:36.866+0000 7f99a486b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:36.866 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:36.866+0000 7f99a486b380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:33:36.866 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:36.866+0000 7f99a486b380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:36.902 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:36.904+0000 7f99a486b380 1 bucket sync caught up with source: 2021-09-02T07:33:36.903 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000006.156.5, , 00000000004.139.5, , 00000000015.151.5, , , , , ] 2021-09-02T07:33:36.903 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000006.156.5, , 00000000004.139.5, , 00000000015.151.5, , , , , ] 2021-09-02T07:33:36.903 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:36.904+0000 7f99a486b380 0 bucket checkpoint complete 2021-09-02T07:33:36.918 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-16 zones={test-zone2, test-zone1} 2021-09-02T07:33:36.925 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:36.928 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:36.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:36.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:36.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:36.930 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:36.934 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:36.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:36.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:36.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:36.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:36.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:37.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:37.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:37.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:37.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:37.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:37.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:37.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:37.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:37.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:37.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:37.826 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-16 zones={test-zone2, test-zone1} 2021-09-02T07:33:37.827 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-16 zones={test-zone3, test-zone1} 2021-09-02T07:33:37.832 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:37.836 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:37.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:37.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:37.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:37.838 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:37.841 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:37.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:37.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:37.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:37.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:37.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:37.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:38.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:38.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:38.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:38.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:38.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:38.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:38.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:38.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:38.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:38.766 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-16 zones={test-zone3, test-zone1} 2021-09-02T07:33:38.767 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-16 zones={test-zone3, test-zone2} 2021-09-02T07:33:38.772 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:38.776 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:38.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:38.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:38.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:38.778 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:38.781 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:38.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:38.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:38.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:38.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:38.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:38.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:38.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:38.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:39.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:39.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:39.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:39.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:39.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:39.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:39.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:39.674 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-16 zones={test-zone3, test-zone2} 2021-09-02T07:33:39.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:39.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:39.676 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:39.729 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:39.730 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:39.753 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:39.755+0000 7f1d3810b380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:39.754 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:39.755+0000 7f1d3810b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:39.754 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:39.755+0000 7f1d3810b380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:33:39.755 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:39.755+0000 7f1d3810b380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:39.793 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:39.794+0000 7f1d3810b380 1 bucket sync caught up with source: 2021-09-02T07:33:39.793 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000011.35.5, , 00000000004.14.5, , 00000000012.27.5, , , , , ] 2021-09-02T07:33:39.794 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000011.35.5, , 00000000004.14.5, , 00000000012.27.5, , , , , ] 2021-09-02T07:33:39.794 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:39.794+0000 7f1d3810b380 0 bucket checkpoint complete 2021-09-02T07:33:39.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:39.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:39.812 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:39.864 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:39.864 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:39.888 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:39.889+0000 7ff3f7d68380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:39.888 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:39.889+0000 7ff3f7d68380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:39.888 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:39.889+0000 7ff3f7d68380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:33:39.889 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:39.889+0000 7ff3f7d68380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:39.921 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:39.923+0000 7ff3f7d68380 1 bucket sync caught up with source: 2021-09-02T07:33:39.922 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000011.35.5, , 00000000004.14.5, , 00000000012.27.5, , , , , ] 2021-09-02T07:33:39.922 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000011.35.5, , 00000000004.14.5, , 00000000012.27.5, , , , , ] 2021-09-02T07:33:39.922 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:39.923+0000 7ff3f7d68380 0 bucket checkpoint complete 2021-09-02T07:33:39.938 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:39.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:39.939 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:39.988 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:39.989 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:40.011 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:40.012+0000 7f8a98cec380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:40.012 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:40.012+0000 7f8a98cec380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:40.012 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:40.012+0000 7f8a98cec380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:33:40.012 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:40.012+0000 7f8a98cec380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:40.056 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:40.057+0000 7f8a98cec380 1 bucket sync caught up with source: 2021-09-02T07:33:40.056 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000011.35.5, , 00000000004.14.5, , 00000000012.27.5, , , , , ] 2021-09-02T07:33:40.057 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000011.35.5, , 00000000004.14.5, , 00000000012.27.5, , , , , ] 2021-09-02T07:33:40.057 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:40.057+0000 7f8a98cec380 0 bucket checkpoint complete 2021-09-02T07:33:40.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:40.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:40.073 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:40.122 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:40.122 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:40.145 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:40.146+0000 7fb4028ba380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:40.145 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:40.146+0000 7fb4028ba380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:40.145 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:40.146+0000 7fb4028ba380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:33:40.145 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:40.146+0000 7fb4028ba380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:40.183 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:40.185+0000 7fb4028ba380 1 bucket sync caught up with source: 2021-09-02T07:33:40.184 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000004.293.5, , 00000000011.205.5, , 00000000013.139.5, , , , , ] 2021-09-02T07:33:40.184 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000004.293.5, , 00000000011.205.5, , 00000000013.139.5, , , , , ] 2021-09-02T07:33:40.184 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:40.185+0000 7fb4028ba380 0 bucket checkpoint complete 2021-09-02T07:33:40.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:40.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:40.201 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:40.251 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:40.251 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:40.272 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:40.274+0000 7f66b4104380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:40.273 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:40.274+0000 7f66b4104380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:40.273 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:40.274+0000 7f66b4104380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:33:40.273 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:40.274+0000 7f66b4104380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:40.303 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:40.305+0000 7f66b4104380 1 bucket sync caught up with source: 2021-09-02T07:33:40.303 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000004.293.5, , 00000000011.205.5, , 00000000013.139.5, , , , , ] 2021-09-02T07:33:40.304 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000004.293.5, , 00000000011.205.5, , 00000000013.139.5, , , , , ] 2021-09-02T07:33:40.304 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:40.305+0000 7f66b4104380 0 bucket checkpoint complete 2021-09-02T07:33:40.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:40.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:40.319 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:40.367 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:40.368 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:40.389 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:40.391+0000 7fb39e0f5380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:40.390 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:40.391+0000 7fb39e0f5380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:40.390 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:40.391+0000 7fb39e0f5380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:33:40.390 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:40.391+0000 7fb39e0f5380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:40.427 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:40.429+0000 7fb39e0f5380 1 bucket sync caught up with source: 2021-09-02T07:33:40.427 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000004.268.5, , 00000000006.167.5, , 00000000015.129.5, , , , , ] 2021-09-02T07:33:40.428 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000004.268.5, , 00000000006.167.5, , 00000000015.129.5, , , , , ] 2021-09-02T07:33:40.428 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:40.429+0000 7fb39e0f5380 0 bucket checkpoint complete 2021-09-02T07:33:40.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:40.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:40.446 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:40.495 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:40.495 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:40.520 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:40.521+0000 7f6e71fbb380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:40.521 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:40.521+0000 7f6e71fbb380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:40.521 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:40.521+0000 7f6e71fbb380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:33:40.521 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:40.521+0000 7f6e71fbb380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:40.551 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:40.552+0000 7f6e71fbb380 1 bucket sync caught up with source: 2021-09-02T07:33:40.551 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000004.268.5, , 00000000006.167.5, , 00000000015.129.5, , , , , ] 2021-09-02T07:33:40.552 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000004.268.5, , 00000000006.167.5, , 00000000015.129.5, , , , , ] 2021-09-02T07:33:40.552 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:40.552+0000 7f6e71fbb380 0 bucket checkpoint complete 2021-09-02T07:33:40.566 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-17 zones={test-zone2, test-zone1} 2021-09-02T07:33:40.572 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:40.576 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:40.576 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:40.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:40.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:40.578 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:40.581 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:40.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:40.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:40.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:40.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:40.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:40.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:40.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:40.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:40.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:40.942 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2021-09-02T07:33:40.944+0000 7f04206f5700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-02T07:33:40.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:41.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:41.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:41.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:41.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:41.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:41.459 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-17 zones={test-zone2, test-zone1} 2021-09-02T07:33:41.460 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-17 zones={test-zone3, test-zone1} 2021-09-02T07:33:41.467 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:41.471 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:41.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:41.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:41.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:41.474 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:41.478 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:41.479 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:41.479 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:41.479 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:41.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:41.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:41.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:41.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:41.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:41.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:41.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:41.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:42.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:42.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:42.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:42.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:42.371 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-17 zones={test-zone3, test-zone1} 2021-09-02T07:33:42.372 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-17 zones={test-zone3, test-zone2} 2021-09-02T07:33:42.378 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:42.382 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:42.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:42.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:42.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:42.384 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:42.388 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:42.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:42.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:42.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:42.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:42.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:42.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:42.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:42.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:42.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:42.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:42.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:43.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:43.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:43.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:43.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:43.371 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-17 zones={test-zone3, test-zone2} 2021-09-02T07:33:43.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:43.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:43.374 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:43.431 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:43.432 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:43.458 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:43.459+0000 7ff764b42380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:43.458 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:43.459+0000 7ff764b42380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:43.458 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:43.459+0000 7ff764b42380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:33:43.458 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:43.459+0000 7ff764b42380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:43.491 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:43.493+0000 7ff764b42380 1 bucket sync caught up with source: 2021-09-02T07:33:43.492 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000011.28.5, , 00000000004.15.5, , 00000000008.30.5, , , , , ] 2021-09-02T07:33:43.492 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000011.28.5, , 00000000004.15.5, , 00000000008.30.5, , , , , ] 2021-09-02T07:33:43.492 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:43.493+0000 7ff764b42380 0 bucket checkpoint complete 2021-09-02T07:33:43.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:43.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:43.511 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:43.567 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:43.567 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:43.593 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:43.594+0000 7fe8fb686380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:43.593 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:43.594+0000 7fe8fb686380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:43.593 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:43.594+0000 7fe8fb686380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:33:43.593 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:43.594+0000 7fe8fb686380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:43.635 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:43.636+0000 7fe8fb686380 1 bucket sync caught up with source: 2021-09-02T07:33:43.636 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000011.28.5, , 00000000004.15.5, , 00000000008.30.5, , , , , ] 2021-09-02T07:33:43.636 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000011.28.5, , 00000000004.15.5, , 00000000008.30.5, , , , , ] 2021-09-02T07:33:43.637 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:43.636+0000 7fe8fb686380 0 bucket checkpoint complete 2021-09-02T07:33:43.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:43.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:43.654 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:43.704 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:43.704 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:43.725 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:43.726+0000 7f7c56796380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:43.725 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:43.726+0000 7f7c56796380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:43.725 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:43.726+0000 7f7c56796380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:33:43.726 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:43.726+0000 7f7c56796380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:43.755 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:43.757+0000 7f7c56796380 1 bucket sync caught up with source: 2021-09-02T07:33:43.756 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000011.28.5, , 00000000004.15.5, , 00000000008.30.5, , , , , ] 2021-09-02T07:33:43.756 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000011.28.5, , 00000000004.15.5, , 00000000008.30.5, , , , , ] 2021-09-02T07:33:43.756 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:43.757+0000 7f7c56796380 0 bucket checkpoint complete 2021-09-02T07:33:43.774 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:43.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:43.775 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:43.826 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:43.827 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:43.849 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:43.850+0000 7f7bc6d33380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:43.849 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:43.850+0000 7f7bc6d33380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:43.849 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:43.850+0000 7f7bc6d33380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:33:43.849 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:43.850+0000 7f7bc6d33380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:43.883 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:43.885+0000 7f7bc6d33380 1 bucket sync caught up with source: 2021-09-02T07:33:43.883 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000004.117.5, , 00000000011.295.5, , 00000000008.282.5, , , , , ] 2021-09-02T07:33:43.884 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000004.117.5, , 00000000011.295.5, , 00000000008.282.5, , , , , ] 2021-09-02T07:33:43.884 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:43.885+0000 7f7bc6d33380 0 bucket checkpoint complete 2021-09-02T07:33:43.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:43.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:43.902 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:43.954 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:43.954 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:43.978 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:43.979+0000 7f48d9d57380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:43.978 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:43.979+0000 7f48d9d57380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:43.978 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:43.979+0000 7f48d9d57380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:33:43.979 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:43.979+0000 7f48d9d57380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:45.019 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:45.020+0000 7f48d9d57380 1 bucket sync caught up with source: 2021-09-02T07:33:45.019 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000004.117.5, , 00000000011.295.5, , 00000000008.282.5, , , , , ] 2021-09-02T07:33:45.020 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000004.117.5, , 00000000011.295.5, , 00000000008.282.5, , , , , ] 2021-09-02T07:33:45.020 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:45.020+0000 7f48d9d57380 0 bucket checkpoint complete 2021-09-02T07:33:45.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:45.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:45.039 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:45.090 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:45.091 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:45.114 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:45.115+0000 7fa5d6d20380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:45.114 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:45.115+0000 7fa5d6d20380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:45.114 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:45.115+0000 7fa5d6d20380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:33:45.115 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:45.115+0000 7fa5d6d20380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:45.150 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:45.151+0000 7fa5d6d20380 1 bucket sync caught up with source: 2021-09-02T07:33:45.151 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, 00000000004.110.5, , 00000000004.230.5, , 00000000017.284.5, , , , , ] 2021-09-02T07:33:45.151 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, 00000000004.110.5, , 00000000004.230.5, , 00000000017.284.5, , , , , ] 2021-09-02T07:33:45.151 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:45.151+0000 7fa5d6d20380 0 bucket checkpoint complete 2021-09-02T07:33:45.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:45.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:45.166 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:45.221 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:45.221 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:45.246 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:45.247+0000 7fa18b354380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:45.247 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:45.247+0000 7fa18b354380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:45.247 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:45.247+0000 7fa18b354380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:33:45.247 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:45.247+0000 7fa18b354380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:45.289 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:45.290+0000 7fa18b354380 1 bucket sync caught up with source: 2021-09-02T07:33:45.289 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, 00000000004.110.5, , 00000000004.230.5, , 00000000017.284.5, , , , , ] 2021-09-02T07:33:45.289 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, 00000000004.110.5, , 00000000004.230.5, , 00000000017.284.5, , , , , ] 2021-09-02T07:33:45.290 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:45.290+0000 7fa18b354380 0 bucket checkpoint complete 2021-09-02T07:33:45.305 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-18 zones={test-zone2, test-zone1} 2021-09-02T07:33:45.312 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:45.316 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:45.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:45.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:45.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:45.318 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:45.321 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:45.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:45.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:45.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:45.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:45.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:45.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:45.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:45.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:45.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:45.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:45.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:45.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:45.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:46.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:46.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:46.218 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-18 zones={test-zone2, test-zone1} 2021-09-02T07:33:46.219 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-18 zones={test-zone3, test-zone1} 2021-09-02T07:33:46.224 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:46.228 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:46.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:46.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:46.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:46.229 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:46.233 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:46.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:46.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:46.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:46.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:46.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:46.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:46.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:46.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:46.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:46.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:46.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:46.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:46.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:46.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:47.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:47.114 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-18 zones={test-zone3, test-zone1} 2021-09-02T07:33:47.115 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-18 zones={test-zone3, test-zone2} 2021-09-02T07:33:47.121 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:47.125 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:47.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:47.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:47.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:47.126 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:47.131 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:47.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-02T07:33:47.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-02T07:33:47.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-02T07:33:47.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:47.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:47.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:47.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:47.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:47.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-02T07:33:47.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:47.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:47.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-02T07:33:47.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:47.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:47.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-02T07:33:48.066 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-18 zones={test-zone3, test-zone2} 2021-09-02T07:33:48.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2021-09-02T07:33:48.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:33:48.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:33:48.095 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:33:48.140 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:48.141 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:48.187 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T07:33:48.201 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:33:48.202 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:33:48.202 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:33:48.202 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:32:10.734248Z" 2021-09-02T07:33:48.203 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:33:48.203 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:33:48.203 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568028.095974_89.1", 2021-09-02T07:33:48.204 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:33:48.095974Z" 2021-09-02T07:33:48.204 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:33:48.204 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:33:48.204 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:33:48.205 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:33:48.205 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:33:48.205 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:33:48.206 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:33:48.206 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:33:48.206 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:33:48.207 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:33:48.208 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T07:33:48.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:33:48.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:33:48.210 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:33:48.258 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:48.258 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:48.335 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:33:48.336 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:33:48.336 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:33:48.336 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:33:48.337 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:33:48.337 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:33:48.337 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:48.337 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:48.338 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:33:48.338 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:48.338 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:33:48.338 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:48.338 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:48.339 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:33:48.339 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:48.339 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:33:48.339 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:48.340 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:33:48.340 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:48.340 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:48.340 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:48.341 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:48.341 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:33:48.341 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:48.341 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:48.342 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568028.079083_84.1", 2021-09-02T07:33:48.342 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:48.342 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:33:48.343 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:48.343 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:33:48.079083Z", 2021-09-02T07:33:48.343 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:48.343 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:48.344 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:48.344 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:48.344 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:33:48.345 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:48.345 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:48.345 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:33:48.345 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:48.346 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:33:48.346 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:48.346 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:33:48.346 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:48.347 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:48.347 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:48.347 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:48.347 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:33:48.347 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:48.348 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:48.348 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:33:48.348 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:48.348 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:33:48.349 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:48.349 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:33:48.349 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:48.349 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:48.350 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:48.350 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:33:48.350 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:48.350 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:33:48.351 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:33:48.351 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:33:48.351 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:48.351 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:33:48.353 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630568028.095974_89.1 target=1_1630568028.079083_84.1 2021-09-02T07:33:53.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:33:53.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:33:53.356 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:33:53.406 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:53.406 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:53.480 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:33:53.480 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:33:53.480 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:33:53.481 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:33:53.481 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:33:53.481 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:33:53.481 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:53.482 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:53.482 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:33:53.482 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:53.482 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:33:53.483 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:53.483 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:53.483 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:33:53.483 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:53.483 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:33:53.484 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:53.484 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:33:53.484 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:53.484 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:53.485 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:53.485 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:53.485 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:33:53.485 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:53.485 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:53.486 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568028.095974_89.1", 2021-09-02T07:33:53.486 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:53.486 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:33:53.486 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:53.486 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:33:48.095974Z", 2021-09-02T07:33:53.487 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:53.487 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:53.487 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:53.487 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:53.488 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:33:53.488 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:53.488 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:53.488 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:33:53.488 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:53.489 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:33:53.489 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:53.489 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:33:53.489 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:53.490 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:53.490 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:53.490 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:53.490 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:33:53.490 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:53.491 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:53.491 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:33:53.491 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:53.491 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:33:53.492 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:53.492 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:33:53.492 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:53.492 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:53.492 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:53.493 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:33:53.493 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:53.493 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:33:53.493 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:33:53.494 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:33:53.494 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:53.494 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:33:53.495 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T07:33:53.496 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T07:33:53.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:33:53.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:33:53.497 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:33:53.546 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:53.546 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:53.610 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:33:53.611 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:33:53.611 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:33:53.611 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:33:53.611 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:33:53.612 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:33:53.612 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:33:53.612 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:33:53.612 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:33:53.612 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:33:53.613 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:33:53.613 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:33:53.613 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:33:53.613 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:33:53.614 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:33:53.614 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:33:53.614 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:33:53.614 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:33:53.615 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:33:53.615 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:33:53.615 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:33:53.615 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:33:53.616 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:33:53.616 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:33:53.616 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:33:53.616 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568028.095974_89.1", 2021-09-02T07:33:53.617 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:33:53.617 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:33:53.617 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:33:53.617 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:33:48.095974Z", 2021-09-02T07:33:53.617 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:33:53.618 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:33:53.618 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:33:53.618 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:33:53.618 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:33:53.619 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:33:53.619 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:33:53.619 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:33:53.619 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:33:53.620 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:33:53.620 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:33:53.620 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:33:53.620 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:33:53.621 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:33:53.621 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:33:53.622 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:33:53.622 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:33:53.622 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:33:53.622 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:33:53.623 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:33:53.623 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:33:53.623 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:33:53.623 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:33:53.623 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:33:53.624 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:33:53.624 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:33:53.624 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:33:53.624 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:33:53.625 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:33:53.625 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:33:53.625 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:33:53.625 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:33:53.626 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:33:53.626 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:33:53.627 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T07:33:53.628 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T07:33:53.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:33:53.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:33:53.629 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:33:53.673 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:53.673 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:53.751 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:33:53.751 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:33:53.751 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:33:53.751 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:33:53.752 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:33:53.752 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:33:53.752 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:53.752 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:53.753 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:33:53.753 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:53.753 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:33:53.753 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:53.753 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:53.754 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:33:53.754 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:53.754 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:33:53.754 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:53.755 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:33:53.755 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:53.755 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:53.755 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:53.756 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:53.756 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:33:53.756 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:53.756 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:53.756 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568028.095974_89.1", 2021-09-02T07:33:53.757 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:53.757 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:33:53.757 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:53.757 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:33:48.095974Z", 2021-09-02T07:33:53.758 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:53.758 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:53.758 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:53.758 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:53.758 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:33:53.759 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:53.759 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:53.759 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:33:53.759 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:53.760 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:33:53.760 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:53.760 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:33:53.760 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:53.760 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:53.761 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:53.761 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:33:53.761 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:33:53.761 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:33:53.762 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:33:53.762 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:33:53.762 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:33:53.762 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:33:53.762 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:33:53.763 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:33:53.763 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:33:53.763 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:53.763 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:53.764 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:33:53.764 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:33:53.764 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:33:53.764 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:33:53.764 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:33:53.765 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:33:53.765 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:33:53.766 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-02T07:33:53.815 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:53.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:53.816 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:53.865 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:53.866 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:53.889 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:53.890+0000 7f6c56200380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:53.889 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:53.890+0000 7f6c56200380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:53.889 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:53.890+0000 7f6c56200380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:33:53.890 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:53.890+0000 7f6c56200380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:53.932 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:33:53.933+0000 7f968e196700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-02T07:33:53.933 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:53.934+0000 7f6c56200380 1 waiting for incremental sync to catch up: 2021-09-02T07:33:53.933 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , , , ] 2021-09-02T07:33:53.933 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , 00000000002.7.10, , , , , ] 2021-09-02T07:33:58.935 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:58.936+0000 7f6c56200380 1 bucket sync caught up with source: 2021-09-02T07:33:58.935 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , 00000000005.13.9, , , , , ] 2021-09-02T07:33:58.936 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , 00000000002.7.10, , , , , ] 2021-09-02T07:33:58.936 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:58.936+0000 7f6c56200380 0 bucket checkpoint complete 2021-09-02T07:33:58.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:58.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:58.958 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:59.014 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:59.014 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:59.039 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.041+0000 7f39c993d380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:59.039 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.041+0000 7f39c993d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:59.040 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.041+0000 7f39c993d380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:33:59.040 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.041+0000 7f39c993d380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:59.081 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.082+0000 7f39c993d380 1 bucket sync caught up with source: 2021-09-02T07:33:59.081 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000005.13.9, , , , , ] 2021-09-02T07:33:59.081 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000005.13.9, , , , , ] 2021-09-02T07:33:59.082 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.082+0000 7f39c993d380 0 bucket checkpoint complete 2021-09-02T07:33:59.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:59.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:59.097 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:59.152 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:59.152 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:59.177 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:59.179+0000 7f19418dc380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:59.177 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:59.179+0000 7f19418dc380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:59.177 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:59.179+0000 7f19418dc380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:33:59.178 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:59.179+0000 7f19418dc380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:59.210 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:59.213+0000 7f19418dc380 1 bucket sync caught up with source: 2021-09-02T07:33:59.211 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , 00000000005.13.9, , , , , ] 2021-09-02T07:33:59.211 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , 00000000005.13.9, , , , , ] 2021-09-02T07:33:59.211 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:59.213+0000 7f19418dc380 0 bucket checkpoint complete 2021-09-02T07:33:59.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:59.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:59.227 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:59.277 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:59.277 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:59.299 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.301+0000 7fea4b040380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:59.300 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.301+0000 7fea4b040380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:59.300 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.301+0000 7fea4b040380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:33:59.300 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.301+0000 7fea4b040380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:59.330 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.332+0000 7fea4b040380 1 bucket sync caught up with source: 2021-09-02T07:33:59.331 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.118.9, , , , , ] 2021-09-02T07:33:59.331 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.118.9, , , , , ] 2021-09-02T07:33:59.331 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.332+0000 7fea4b040380 0 bucket checkpoint complete 2021-09-02T07:33:59.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:59.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:59.347 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:59.396 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:59.396 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:59.419 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.420+0000 7ff8f2fe0380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:59.419 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.420+0000 7ff8f2fe0380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:59.419 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.420+0000 7ff8f2fe0380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:33:59.419 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.420+0000 7ff8f2fe0380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:59.451 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.453+0000 7ff8f2fe0380 1 bucket sync caught up with source: 2021-09-02T07:33:59.452 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.118.9, , , , , ] 2021-09-02T07:33:59.452 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.118.9, , , , , ] 2021-09-02T07:33:59.452 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.453+0000 7ff8f2fe0380 0 bucket checkpoint complete 2021-09-02T07:33:59.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:59.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:59.472 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:59.521 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:59.521 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:59.543 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.545+0000 7ffa17105380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:59.543 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.545+0000 7ffa17105380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:59.544 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.545+0000 7ffa17105380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:33:59.544 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.545+0000 7ffa17105380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:59.578 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.580+0000 7ffa17105380 1 bucket sync caught up with source: 2021-09-02T07:33:59.578 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.117.9, , , , , ] 2021-09-02T07:33:59.578 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.117.9, , , , , ] 2021-09-02T07:33:59.579 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:33:59.580+0000 7ffa17105380 0 bucket checkpoint complete 2021-09-02T07:33:59.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:59.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:33:59.594 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:33:59.645 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:33:59.645 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:33:59.667 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:59.669+0000 7f85cb092380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:33:59.667 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:59.669+0000 7f85cb092380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:33:59.667 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:59.669+0000 7f85cb092380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:33:59.668 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:59.669+0000 7f85cb092380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:33:59.718 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:59.719+0000 7f85cb092380 1 bucket sync caught up with source: 2021-09-02T07:33:59.718 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , 00000000004.117.9, , , , , ] 2021-09-02T07:33:59.719 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , 00000000004.117.9, , , , , ] 2021-09-02T07:33:59.719 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:33:59.719+0000 7f85cb092380 0 bucket checkpoint complete 2021-09-02T07:33:59.733 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-19 zones={test-zone2, test-zone1} 2021-09-02T07:33:59.739 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:33:59.783 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:59.784 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:33:59.789 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:33:59.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:59.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:59.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:33:59.962 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-19 zones={test-zone2, test-zone1} 2021-09-02T07:33:59.963 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-19 zones={test-zone3, test-zone1} 2021-09-02T07:33:59.969 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:34:00.015 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:34:00.016 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:34:00.019 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:34:00.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:34:00.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:34:00.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:34:00.187 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-19 zones={test-zone3, test-zone1} 2021-09-02T07:34:00.187 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-19 zones={test-zone3, test-zone2} 2021-09-02T07:34:00.193 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:34:00.239 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:34:00.240 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:34:00.244 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:34:00.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:34:00.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:34:00.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:34:00.414 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-19 zones={test-zone3, test-zone2} 2021-09-02T07:34:02.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:02.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:02.009 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:02.064 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:02.065 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:02.087 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:02.088+0000 7fb30aa28380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:02.087 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:02.088+0000 7fb30aa28380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:02.087 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:02.088+0000 7fb30aa28380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:34:02.088 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:02.088+0000 7fb30aa28380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:02.120 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:02.122+0000 7fb30aa28380 1 bucket sync caught up with source: 2021-09-02T07:34:02.121 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , 00000000005.13.9, , , 00000000056.163.11, , ] 2021-09-02T07:34:02.121 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , 00000000005.13.9, , , 00000000056.163.11, , ] 2021-09-02T07:34:02.122 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:02.122+0000 7fb30aa28380 0 bucket checkpoint complete 2021-09-02T07:34:02.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:02.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:02.140 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:02.194 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:02.194 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:02.215 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:02.217+0000 7f4e47685380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:02.215 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:02.217+0000 7f4e47685380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:02.216 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:02.217+0000 7f4e47685380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:34:02.216 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:02.217+0000 7f4e47685380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:02.247 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:02.249+0000 7f4e47685380 1 waiting for incremental sync to catch up: 2021-09-02T07:34:02.248 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000005.13.9, , , 00000000058.168.8, , ] 2021-09-02T07:34:02.248 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000005.13.9, , , 00000000060.172.9, , ] 2021-09-02T07:34:07.249 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:07.250+0000 7f4e47685380 1 bucket sync caught up with source: 2021-09-02T07:34:07.249 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000005.13.9, , , 00000000060.172.9, , ] 2021-09-02T07:34:07.249 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000005.13.9, , , 00000000060.172.9, , ] 2021-09-02T07:34:07.250 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:07.250+0000 7f4e47685380 0 bucket checkpoint complete 2021-09-02T07:34:07.268 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:07.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:07.270 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:07.325 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:07.326 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:07.350 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:07.353+0000 7ff364f68380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:07.351 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:07.353+0000 7ff364f68380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:07.351 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:07.353+0000 7ff364f68380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:34:07.351 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:07.353+0000 7ff364f68380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:07.398 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:07.400+0000 7ff364f68380 1 bucket sync caught up with source: 2021-09-02T07:34:07.398 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , 00000000005.13.9, , , 00000000060.172.9, , ] 2021-09-02T07:34:07.399 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , 00000000005.13.9, , , 00000000060.172.9, , ] 2021-09-02T07:34:07.399 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:07.400+0000 7ff364f68380 0 bucket checkpoint complete 2021-09-02T07:34:07.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:07.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:07.415 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:07.471 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:07.471 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:07.495 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:07.497+0000 7fab161c1380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:07.496 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:07.497+0000 7fab161c1380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:07.496 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:07.497+0000 7fab161c1380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:34:07.496 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:07.497+0000 7fab161c1380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:07.535 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:07.536+0000 7fab161c1380 1 bucket sync caught up with source: 2021-09-02T07:34:07.535 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.118.9, , , 00000000056.455.9, , ] 2021-09-02T07:34:07.535 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.118.9, , , 00000000056.455.9, , ] 2021-09-02T07:34:07.536 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:07.536+0000 7fab161c1380 0 bucket checkpoint complete 2021-09-02T07:34:07.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:07.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:07.554 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:07.603 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:07.603 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:07.625 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:07.626+0000 7f84abec9380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:07.625 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:07.626+0000 7f84abec9380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:07.626 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:07.626+0000 7f84abec9380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:34:07.626 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:07.626+0000 7f84abec9380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:07.661 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:07.663+0000 7f84abec9380 1 bucket sync caught up with source: 2021-09-02T07:34:07.661 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.118.9, , , 00000000056.455.9, , ] 2021-09-02T07:34:07.662 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.118.9, , , 00000000056.455.9, , ] 2021-09-02T07:34:07.662 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:07.663+0000 7f84abec9380 0 bucket checkpoint complete 2021-09-02T07:34:07.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:07.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:07.678 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:07.726 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:07.727 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:07.749 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:07.751+0000 7f44c597d380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:07.750 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:07.751+0000 7f44c597d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:07.750 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:07.751+0000 7f44c597d380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:34:07.750 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:07.751+0000 7f44c597d380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:07.785 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:07.787+0000 7f44c597d380 1 bucket sync caught up with source: 2021-09-02T07:34:07.785 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , 00000000004.117.9, , , 00000000058.460.9, , ] 2021-09-02T07:34:07.786 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , 00000000004.117.9, , , 00000000058.460.9, , ] 2021-09-02T07:34:07.786 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:07.787+0000 7f44c597d380 0 bucket checkpoint complete 2021-09-02T07:34:07.801 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:07.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:07.803 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:07.852 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:07.853 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:07.874 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:07.876+0000 7f99c50ae380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:07.875 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:07.876+0000 7f99c50ae380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:07.875 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:07.877+0000 7f99c50ae380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:34:07.875 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:07.877+0000 7f99c50ae380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:07.918 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:07.920+0000 7f99c50ae380 1 bucket sync caught up with source: 2021-09-02T07:34:07.919 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , 00000000004.117.9, , , 00000000058.460.9, , ] 2021-09-02T07:34:07.919 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , 00000000004.117.9, , , 00000000058.460.9, , ] 2021-09-02T07:34:07.920 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:07.920+0000 7f99c50ae380 0 bucket checkpoint complete 2021-09-02T07:34:07.936 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-19 zones={test-zone2, test-zone1} 2021-09-02T07:34:07.943 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:34:07.948 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:34:07.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.953 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.962 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:34:07.967 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:34:07.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.975 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.975 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:07.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:34:07.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:34:08.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:34:08.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:08.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:08.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:08.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:08.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:08.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:08.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:08.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:08.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:08.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:08.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:09.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:09.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:09.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:09.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:09.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:09.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:09.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:09.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:09.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:09.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:09.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:09.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:10.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:10.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:10.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:10.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:10.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:10.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:10.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:10.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:10.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:10.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:10.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:11.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:11.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:11.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:11.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:11.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:11.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:11.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:11.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:11.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:11.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:11.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:12.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:12.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:12.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:12.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:12.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:12.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:12.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:12.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:12.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:12.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:12.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:13.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:13.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:13.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:13.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:13.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:13.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:13.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:13.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:13.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:13.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:13.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:13.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:14.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:14.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:14.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:14.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:14.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:14.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:14.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:14.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:14.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:14.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:14.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:15.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:15.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:15.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:15.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:15.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:15.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:15.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:15.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:15.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:15.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:15.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:16.011 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-19 zones={test-zone2, test-zone1} 2021-09-02T07:34:16.011 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-19 zones={test-zone3, test-zone1} 2021-09-02T07:34:16.018 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:34:16.024 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:34:16.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.035 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.038 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.038 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:34:16.044 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:34:16.045 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.045 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.050 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.050 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.056 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.057 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.058 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.059 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.059 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:16.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:34:16.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:34:16.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:34:16.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:16.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:16.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:16.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:16.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:16.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:16.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:16.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:16.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:17.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:17.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:17.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:17.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:17.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:17.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:17.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:17.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:17.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:17.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:17.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:17.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:18.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:18.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:18.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:18.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:18.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:18.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:18.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:18.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:18.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:18.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:18.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:18.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:19.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:19.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:19.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:19.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:19.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:19.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:19.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:19.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:19.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:19.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:19.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:20.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:20.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:20.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:20.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:20.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:20.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:20.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:20.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:20.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:20.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:20.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:21.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:21.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:21.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:21.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:21.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:21.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:21.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:21.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:21.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:21.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:21.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:22.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:22.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:22.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:22.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:22.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:22.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:22.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:22.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:22.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:22.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:22.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:22.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:23.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:23.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:23.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:23.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:23.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:23.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:23.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:23.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:23.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:23.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:23.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:23.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:24.078 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-19 zones={test-zone3, test-zone1} 2021-09-02T07:34:24.079 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-19 zones={test-zone3, test-zone2} 2021-09-02T07:34:24.085 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:34:24.091 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:34:24.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.107 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:34:24.113 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-02T07:34:24.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:24.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:34:24.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:34:24.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-02T07:34:24.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:24.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:24.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:24.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:24.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:24.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:24.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:24.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:24.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:25.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:25.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:25.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:25.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:25.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:25.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:25.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:25.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:25.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:25.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:25.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:25.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:26.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:26.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:26.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:26.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:26.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:26.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:26.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:26.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:26.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:26.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:26.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:27.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:27.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:27.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:27.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:27.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:27.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:27.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:27.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:27.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:27.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:27.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:27.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:28.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:28.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:28.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:28.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:28.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:28.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:28.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:28.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:28.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:28.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:28.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:28.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:29.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:29.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:29.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:29.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:29.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:29.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:29.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:29.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:29.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:29.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:29.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:29.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:30.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:30.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:30.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:30.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:30.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:30.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:30.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:30.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:30.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:30.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:30.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:31.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:31.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:31.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:31.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:31.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:31.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:31.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:31.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:31.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:31.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:31.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:32.079 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-19 zones={test-zone3, test-zone2} 2021-09-02T07:34:32.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:32.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:32.081 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:32.129 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:32.130 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:32.205 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:34:32.205 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:32.206 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-02T07:34:32.206 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:34:03.766866Z" 2021-09-02T07:34:32.206 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:32.206 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:32.207 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-02T07:34:32.207 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:34:03.768742Z" 2021-09-02T07:34:32.207 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:32.207 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:32.208 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:32.208 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:34:32.208 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:32.208 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:32.209 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-02T07:34:32.209 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:33:41.763460Z" 2021-09-02T07:34:32.209 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:32.209 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:34:32.211 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-09-02T07:34:32.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:32.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:32.212 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:32.262 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:32.263 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:32.334 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:34:32.335 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:34:32.335 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:34:32.335 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:34:32.336 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:34:32.336 INFO:teuthology.orchestra.run.smithi197.stdout: "instance_id": 938678123907490605 2021-09-02T07:34:32.336 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:32.336 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:34:32.336 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:32.337 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:34:32.337 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:32.337 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "incremental-sync", 2021-09-02T07:34:32.337 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-02T07:34:32.338 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:32.338 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:32.338 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:32.338 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:34:03.766866Z" 2021-09-02T07:34:32.339 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:32.339 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:32.339 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:32.339 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:34:32.340 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:32.340 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "incremental-sync", 2021-09-02T07:34:32.340 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-02T07:34:32.340 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:32.341 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:32.341 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:32.341 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:34:03.768742Z" 2021-09-02T07:34:32.341 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:32.341 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:32.342 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:32.342 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:34:32.342 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:32.342 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "incremental-sync", 2021-09-02T07:34:32.343 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:32.343 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:32.343 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:32.343 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:32.344 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000" 2021-09-02T07:34:32.344 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:32.344 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:32.344 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:32.344 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:34:32.345 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:32.345 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "incremental-sync", 2021-09-02T07:34:32.345 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-02T07:34:32.345 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:32.346 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:32.346 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:32.346 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:33:41.763460Z" 2021-09-02T07:34:32.346 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:32.347 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:32.347 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:34:32.347 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:32.347 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:34:32.347 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 0, 2021-09-02T07:34:32.348 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 0 2021-09-02T07:34:32.348 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:32.348 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:34:32.349 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-09-02T07:34:32.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:32.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:32.351 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:32.393 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:32.393 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:32.464 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:34:32.465 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:32.465 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-02T07:34:32.465 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:34:03.766866Z" 2021-09-02T07:34:32.466 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:32.466 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:32.466 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-02T07:34:32.466 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:34:03.768742Z" 2021-09-02T07:34:32.467 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:32.467 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:32.467 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:32.467 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:34:32.467 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:32.468 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:32.468 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-02T07:34:32.468 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:33:41.763460Z" 2021-09-02T07:34:32.468 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:32.469 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:34:32.470 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-09-02T07:34:32.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:32.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:32.471 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:32.553 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:32.554 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:32.614 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:34:32.614 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:34:32.615 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:34:32.615 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:34:32.615 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:34:32.615 INFO:teuthology.orchestra.run.smithi175.stdout: "instance_id": 8716742107351376726 2021-09-02T07:34:32.615 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:32.616 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:34:32.616 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:32.616 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:34:32.616 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:32.617 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2021-09-02T07:34:32.617 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-02T07:34:32.617 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:32.617 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:32.617 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:32.618 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:34:03.766866Z" 2021-09-02T07:34:32.618 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:32.618 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:32.618 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:32.619 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:34:32.619 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:32.619 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2021-09-02T07:34:32.619 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-02T07:34:32.619 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:32.620 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:32.620 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:32.620 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:34:03.768742Z" 2021-09-02T07:34:32.620 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:32.621 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:32.621 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:32.621 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:34:32.621 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:32.621 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2021-09-02T07:34:32.622 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:32.622 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:32.622 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:32.622 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:32.623 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000" 2021-09-02T07:34:32.623 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:32.623 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:32.623 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:32.623 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:34:32.624 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:32.624 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2021-09-02T07:34:32.624 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-02T07:34:32.624 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:32.625 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:32.625 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:32.625 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:33:41.763460Z" 2021-09-02T07:34:32.625 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:32.625 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:32.626 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:34:32.626 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:32.626 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:34:32.626 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 0, 2021-09-02T07:34:32.627 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 0 2021-09-02T07:34:32.627 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:32.627 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:34:32.628 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-09-02T07:34:32.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:32.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:32.630 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:32.710 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:32.710 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:32.784 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:34:32.784 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:32.784 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-02T07:34:32.784 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:34:03.766866Z" 2021-09-02T07:34:32.785 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:32.785 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:32.785 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-02T07:34:32.785 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:34:03.768742Z" 2021-09-02T07:34:32.786 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:32.786 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:32.786 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:32.786 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:34:32.786 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:32.787 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:32.787 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-02T07:34:32.787 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:33:41.763460Z" 2021-09-02T07:34:32.787 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:32.788 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:34:32.789 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-09-02T07:34:32.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:32.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:32.791 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:32.839 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:32.839 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:32.904 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:34:32.904 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:34:32.904 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:34:32.904 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:34:32.905 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:34:32.905 INFO:teuthology.orchestra.run.smithi197.stdout: "instance_id": 8057724551400926128 2021-09-02T07:34:32.905 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:32.905 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:34:32.906 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:32.906 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:34:32.906 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:32.906 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "incremental-sync", 2021-09-02T07:34:32.907 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-02T07:34:32.907 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:32.907 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:32.907 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:32.907 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:34:03.766866Z" 2021-09-02T07:34:32.908 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:32.908 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:32.908 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:32.908 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:34:32.909 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:32.909 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "incremental-sync", 2021-09-02T07:34:32.909 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-02T07:34:32.909 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:32.909 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:32.910 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:32.910 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:34:03.768742Z" 2021-09-02T07:34:32.910 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:32.911 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:32.911 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:32.911 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:34:32.911 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:32.911 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "incremental-sync", 2021-09-02T07:34:32.912 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:32.912 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:32.912 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:32.913 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:32.913 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000" 2021-09-02T07:34:32.913 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:32.913 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:32.914 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:32.914 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:34:32.914 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:32.914 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "incremental-sync", 2021-09-02T07:34:32.915 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-02T07:34:32.915 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:32.915 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:32.915 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:32.916 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:33:41.763460Z" 2021-09-02T07:34:32.916 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:32.916 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:32.916 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:34:32.917 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:32.917 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:34:32.917 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 0, 2021-09-02T07:34:32.917 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 0 2021-09-02T07:34:32.917 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:32.918 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:34:32.919 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-09-02T07:34:32.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:32.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:32.920 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:32.961 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:32.962 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:33.040 INFO:teuthology.orchestra.run.smithi197.stdout:[ 2021-09-02T07:34:33.041 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:33.041 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-02T07:34:33.041 INFO:teuthology.orchestra.run.smithi197.stdout: "last_update": "2021-09-02T07:34:03.755956Z" 2021-09-02T07:34:33.041 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:33.042 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:33.042 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-02T07:34:33.042 INFO:teuthology.orchestra.run.smithi197.stdout: "last_update": "2021-09-02T07:34:03.757901Z" 2021-09-02T07:34:33.042 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:33.043 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:33.043 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:33.043 INFO:teuthology.orchestra.run.smithi197.stdout: "last_update": "0.000000" 2021-09-02T07:34:33.043 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:33.044 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:33.044 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-02T07:34:33.044 INFO:teuthology.orchestra.run.smithi197.stdout: "last_update": "2021-09-02T07:33:41.754230Z" 2021-09-02T07:34:33.044 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:33.044 INFO:teuthology.orchestra.run.smithi197.stdout:] 2021-09-02T07:34:33.046 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-09-02T07:34:33.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:33.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:33.047 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:33.093 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:33.093 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:33.161 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:34:33.162 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:34:33.162 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:34:33.162 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:34:33.162 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:34:33.163 INFO:teuthology.orchestra.run.smithi175.stdout: "instance_id": 254626818017318540 2021-09-02T07:34:33.163 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.163 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:34:33.164 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:33.164 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:34:33.164 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:33.165 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2021-09-02T07:34:33.165 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-02T07:34:33.165 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:33.165 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:33.165 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:33.166 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:34:03.755956Z" 2021-09-02T07:34:33.166 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:33.166 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.166 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:33.167 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:34:33.167 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:33.167 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2021-09-02T07:34:33.168 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-02T07:34:33.168 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:33.169 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:33.169 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:33.169 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:34:03.757901Z" 2021-09-02T07:34:33.170 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:33.171 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.171 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:33.171 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:34:33.171 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:33.172 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2021-09-02T07:34:33.172 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:33.172 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:33.172 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:33.173 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:33.173 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000" 2021-09-02T07:34:33.174 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:33.174 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.175 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:33.175 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:34:33.175 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:33.175 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2021-09-02T07:34:33.176 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-02T07:34:33.176 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:33.176 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:33.176 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:33.176 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:33:41.754230Z" 2021-09-02T07:34:33.177 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:33.177 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:33.177 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:34:33.177 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.178 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:34:33.178 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 0, 2021-09-02T07:34:33.178 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 0 2021-09-02T07:34:33.178 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:33.179 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:34:33.180 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-09-02T07:34:33.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:33.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:33.182 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:33.224 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:33.225 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:33.301 INFO:teuthology.orchestra.run.smithi197.stdout:[ 2021-09-02T07:34:33.302 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:33.302 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-02T07:34:33.302 INFO:teuthology.orchestra.run.smithi197.stdout: "last_update": "2021-09-02T07:34:03.755956Z" 2021-09-02T07:34:33.302 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:33.303 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:33.303 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-02T07:34:33.303 INFO:teuthology.orchestra.run.smithi197.stdout: "last_update": "2021-09-02T07:34:03.757901Z" 2021-09-02T07:34:33.303 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:33.304 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:33.304 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:33.304 INFO:teuthology.orchestra.run.smithi197.stdout: "last_update": "0.000000" 2021-09-02T07:34:33.304 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:33.305 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:33.305 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-02T07:34:33.305 INFO:teuthology.orchestra.run.smithi197.stdout: "last_update": "2021-09-02T07:33:41.754230Z" 2021-09-02T07:34:33.305 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:33.305 INFO:teuthology.orchestra.run.smithi197.stdout:] 2021-09-02T07:34:33.307 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-09-02T07:34:33.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:33.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:33.310 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:33.354 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:33.354 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:33.424 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:34:33.425 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:34:33.425 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:34:33.425 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:34:33.426 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:34:33.426 INFO:teuthology.orchestra.run.smithi175.stdout: "instance_id": 5738396314165528603 2021-09-02T07:34:33.427 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.427 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:34:33.427 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:33.427 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:34:33.428 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:33.428 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2021-09-02T07:34:33.428 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-02T07:34:33.428 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:33.428 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:33.429 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:33.429 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:34:03.755956Z" 2021-09-02T07:34:33.429 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:33.429 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.430 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:33.430 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:34:33.430 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:33.430 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2021-09-02T07:34:33.431 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-02T07:34:33.431 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:33.431 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:33.431 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:33.432 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:34:03.757901Z" 2021-09-02T07:34:33.432 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:33.432 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.432 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:33.432 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:34:33.433 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:33.433 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2021-09-02T07:34:33.433 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:33.433 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:33.434 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:33.434 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:33.434 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000" 2021-09-02T07:34:33.434 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:33.435 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.435 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:33.435 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:34:33.435 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:33.435 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2021-09-02T07:34:33.437 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-02T07:34:33.437 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:33.437 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:33.438 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:33.438 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:33:41.754230Z" 2021-09-02T07:34:33.438 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:33.438 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:33.438 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:34:33.439 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.439 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:34:33.439 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 0, 2021-09-02T07:34:33.439 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 0 2021-09-02T07:34:33.440 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:33.440 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:34:33.441 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-09-02T07:34:33.442 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:33.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:33.443 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:33.483 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:33.483 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:33.553 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:34:33.553 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:33.554 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-02T07:34:33.554 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:34:03.819552Z" 2021-09-02T07:34:33.554 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.554 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:33.555 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-02T07:34:33.555 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:34:03.821428Z" 2021-09-02T07:34:33.555 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.555 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:33.556 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:33.556 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:34:33.556 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.556 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:33.556 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-02T07:34:33.557 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:33:41.817642Z" 2021-09-02T07:34:33.557 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:33.557 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:34:33.558 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-09-02T07:34:33.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:33.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:33.560 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:33.641 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:33.642 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:33.705 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:34:33.705 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:34:33.705 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:34:33.705 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:34:33.706 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:34:33.706 INFO:teuthology.orchestra.run.smithi175.stdout: "instance_id": 1470290001344818335 2021-09-02T07:34:33.706 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.706 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:34:33.707 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:33.707 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:34:33.707 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:33.707 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2021-09-02T07:34:33.708 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-02T07:34:33.708 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:33.708 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:33.708 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:33.708 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:34:03.819552Z" 2021-09-02T07:34:33.709 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:33.709 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.709 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:33.709 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:34:33.709 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:33.710 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2021-09-02T07:34:33.710 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-02T07:34:33.710 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:33.710 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:33.711 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:33.711 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:34:03.821428Z" 2021-09-02T07:34:33.711 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:33.711 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.712 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:33.712 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:34:33.712 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:33.712 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2021-09-02T07:34:33.713 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:33.713 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:33.713 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:33.713 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:33.713 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000" 2021-09-02T07:34:33.714 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:33.714 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.714 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:33.714 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:34:33.715 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:33.715 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "incremental-sync", 2021-09-02T07:34:33.715 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-02T07:34:33.715 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:33.716 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:33.716 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:33.716 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:33:41.817642Z" 2021-09-02T07:34:33.716 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:33.716 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:33.717 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:34:33.717 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.717 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:34:33.717 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 0, 2021-09-02T07:34:33.718 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 0 2021-09-02T07:34:33.718 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:33.718 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:34:33.719 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-09-02T07:34:33.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:33.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:33.720 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:33.799 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:33.800 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:33.875 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:34:33.875 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:33.875 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-02T07:34:33.876 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:34:03.819552Z" 2021-09-02T07:34:33.876 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.876 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:33.876 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-02T07:34:33.877 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:34:03.821428Z" 2021-09-02T07:34:33.877 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.877 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:33.877 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:33.878 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:34:33.878 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:33.878 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:33.878 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-02T07:34:33.879 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:33:41.817642Z" 2021-09-02T07:34:33.879 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:33.879 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:34:33.880 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-09-02T07:34:33.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:33.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:33.882 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:33.926 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:33.926 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:33.997 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:34:33.998 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:34:33.998 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:34:33.998 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:34:33.998 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:34:33.999 INFO:teuthology.orchestra.run.smithi197.stdout: "instance_id": 17234276568626628064 2021-09-02T07:34:33.999 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:33.999 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:34:33.999 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:34.000 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:34:34.000 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:34.000 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "incremental-sync", 2021-09-02T07:34:34.000 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-02T07:34:34.001 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:34.001 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:34.001 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:34.001 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:34:03.819552Z" 2021-09-02T07:34:34.002 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:34.002 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:34.002 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:34.002 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:34:34.002 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:34.003 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "incremental-sync", 2021-09-02T07:34:34.003 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-02T07:34:34.003 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:34.003 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:34.004 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:34.004 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:34:03.821428Z" 2021-09-02T07:34:34.004 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:34.004 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:34.005 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:34.005 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:34:34.005 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:34.005 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "incremental-sync", 2021-09-02T07:34:34.005 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:34.006 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:34.006 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:34.006 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:34.006 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000" 2021-09-02T07:34:34.007 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:34.007 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:34.007 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:34.007 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:34:34.008 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:34.008 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "incremental-sync", 2021-09-02T07:34:34.008 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-02T07:34:34.008 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:34.008 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:34.009 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:34.009 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:33:41.817642Z" 2021-09-02T07:34:34.009 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:34.009 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:34.010 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:34:34.010 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:34.010 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:34:34.010 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 0, 2021-09-02T07:34:34.011 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 0 2021-09-02T07:34:34.011 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:34.011 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:34:34.012 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-09-02T07:34:34.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2021-09-02T07:34:34.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:34.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:34.035 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:34.076 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:34.077 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:34.120 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T07:34:34.135 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:34:34.135 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:34.136 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:34:34.136 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:32:10.734248Z" 2021-09-02T07:34:34.136 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:34.136 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:34.137 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568074.033650_95.1", 2021-09-02T07:34:34.137 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:34:34.033650Z" 2021-09-02T07:34:34.137 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:34.137 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:34.137 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:34.138 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:34:34.138 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:34.138 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:34.138 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:34.139 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:34:34.139 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:34.139 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:34:34.141 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T07:34:34.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:34.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:34.142 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:34.186 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:34.186 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:34.257 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:34:34.258 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:34:34.258 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:34:34.258 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:34:34.258 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:34:34.259 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:34:34.259 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:34.259 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:34.259 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:34:34.260 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:34.260 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:34:34.260 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:34.260 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:34.260 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:34:34.261 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:34.261 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:34:34.261 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:34.261 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:34:34.262 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:34.262 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:34.262 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:34.262 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:34.263 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:34:34.263 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:34.263 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:34.263 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568074.025926_92.1", 2021-09-02T07:34:34.263 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:34.264 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:34.264 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:34.264 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:34:34.025926Z", 2021-09-02T07:34:34.264 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:34.265 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:34.265 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:34.265 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:34.266 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:34:34.266 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:34.266 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:34.267 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:34.267 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:34.267 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:34.267 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:34.268 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:34:34.268 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:34.268 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:34.268 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:34.268 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:34.269 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:34:34.269 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:34.269 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:34.269 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:34.270 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:34.270 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:34.270 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:34.270 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:34:34.270 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:34.271 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:34.271 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:34.271 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:34:34.271 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:34.272 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:34:34.272 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:34:34.272 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:34:34.272 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:34.273 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:34:34.274 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630568074.033650_95.1 target=1_1630568074.025926_92.1 2021-09-02T07:34:39.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:39.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:39.277 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:39.327 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:39.327 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:39.409 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:34:39.410 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:34:39.410 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:34:39.410 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:34:39.411 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:34:39.411 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:34:39.411 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:39.412 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:39.412 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:34:39.412 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:39.412 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:34:39.413 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:39.413 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:39.413 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:34:39.413 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:39.413 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:34:39.414 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:39.414 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:34:39.414 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:39.414 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:39.415 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:39.415 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:39.415 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:34:39.415 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:39.416 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:39.416 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568074.033650_95.1", 2021-09-02T07:34:39.416 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:39.416 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:39.417 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:39.417 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:34:34.033650Z", 2021-09-02T07:34:39.417 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:39.417 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:39.417 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:39.418 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:39.418 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:34:39.418 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:39.418 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:39.419 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:39.419 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:39.419 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:39.419 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:39.420 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:34:39.420 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:39.420 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:39.420 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:39.421 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:39.421 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:34:39.421 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:39.421 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:39.421 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:39.422 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:39.422 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:39.422 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:39.422 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:34:39.423 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:39.423 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:39.423 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:39.423 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:34:39.424 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:39.424 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:34:39.424 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:34:39.424 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:34:39.424 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:39.425 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:34:39.426 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T07:34:39.426 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T07:34:39.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:39.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:39.428 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:39.477 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:39.477 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:39.546 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:34:39.546 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:34:39.546 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:34:39.547 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:34:39.547 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:34:39.547 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:34:39.547 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:34:39.548 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:39.548 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:34:39.548 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:39.548 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:34:39.548 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:39.549 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:34:39.549 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:34:39.549 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:39.549 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:34:39.550 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:39.550 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:34:39.550 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:34:39.550 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:39.551 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:39.551 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:39.551 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:34:39.551 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:39.551 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:34:39.552 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568074.033650_95.1", 2021-09-02T07:34:39.552 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:39.552 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:39.552 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:39.553 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:34:34.033650Z", 2021-09-02T07:34:39.553 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:34:39.553 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:39.553 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:39.553 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:39.554 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:34:39.554 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:39.554 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:34:39.554 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:39.555 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:39.555 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:39.555 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:39.555 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:34:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:34:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:39.556 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:34:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:34:39.557 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:39.558 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:39.558 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:39.558 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:39.558 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:34:39.559 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:34:39.559 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:39.559 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:39.559 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:34:39.559 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:39.560 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:34:39.560 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:34:39.560 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:34:39.560 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:39.561 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:34:39.562 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T07:34:39.562 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T07:34:39.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:39.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:39.564 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:39.607 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:39.608 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:39.686 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:34:39.686 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:34:39.687 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:34:39.687 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:34:39.687 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:34:39.687 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:34:39.688 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:39.688 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:39.688 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:34:39.688 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:39.688 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:34:39.689 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:39.689 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:39.689 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:34:39.689 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:39.690 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:34:39.690 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:39.690 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:34:39.690 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:39.691 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:39.691 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:39.691 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:39.691 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:34:39.692 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:39.692 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:39.692 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568074.033650_95.1", 2021-09-02T07:34:39.692 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:39.693 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:39.693 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:39.693 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:34:34.033650Z", 2021-09-02T07:34:39.693 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:39.693 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:39.694 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:39.694 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:39.694 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:34:39.694 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:39.695 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:39.695 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:39.695 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:39.695 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:39.696 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:39.696 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:34:39.696 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:39.696 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:39.697 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:39.697 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:39.697 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:34:39.697 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:39.698 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:39.698 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:39.698 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:39.698 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:39.698 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:39.699 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:34:39.699 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:39.699 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:39.700 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:39.700 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:34:39.700 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:39.700 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:34:39.701 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:34:39.701 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:34:39.701 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:39.701 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:34:39.704 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-02T07:34:39.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:39.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:39.752 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:39.802 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:39.802 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:39.824 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:39.826+0000 7fe49ea37380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:39.825 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:39.826+0000 7fe49ea37380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:39.825 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:39.826+0000 7fe49ea37380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:34:39.825 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:39.826+0000 7fe49ea37380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:39.863 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:39.864+0000 7fe49ea37380 1 bucket sync caught up with source: 2021-09-02T07:34:39.864 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000001.11.5, , ] 2021-09-02T07:34:39.864 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000001.11.5, , ] 2021-09-02T07:34:39.864 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:39.864+0000 7fe49ea37380 0 bucket checkpoint complete 2021-09-02T07:34:39.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:39.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:39.883 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:39.933 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:39.933 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:39.958 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:39.959+0000 7fda03874380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:39.958 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:39.959+0000 7fda03874380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:39.959 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:39.959+0000 7fda03874380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:34:39.959 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:39.959+0000 7fda03874380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:39.989 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:39.991+0000 7fda03874380 1 bucket sync caught up with source: 2021-09-02T07:34:39.990 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.11.5, , ] 2021-09-02T07:34:39.990 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.11.5, , ] 2021-09-02T07:34:39.990 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:39.991+0000 7fda03874380 0 bucket checkpoint complete 2021-09-02T07:34:40.006 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:40.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:40.007 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:40.058 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:40.059 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:40.084 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:40.081+0000 7f5a85fd3380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:40.085 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:40.081+0000 7f5a85fd3380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:40.085 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:40.081+0000 7f5a85fd3380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:34:40.085 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:40.081+0000 7f5a85fd3380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:40.117 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:40.118+0000 7f5a85fd3380 1 bucket sync caught up with source: 2021-09-02T07:34:40.117 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000001.11.5, , ] 2021-09-02T07:34:40.117 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000001.11.5, , ] 2021-09-02T07:34:40.118 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:40.118+0000 7f5a85fd3380 0 bucket checkpoint complete 2021-09-02T07:34:40.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:40.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:40.133 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:40.182 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:40.183 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:40.205 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:40.206+0000 7fd3d77b4380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:40.205 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:40.206+0000 7fd3d77b4380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:40.205 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:40.206+0000 7fd3d77b4380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:34:40.206 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:40.206+0000 7fd3d77b4380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:40.237 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:40.238+0000 7fd3d77b4380 1 bucket sync caught up with source: 2021-09-02T07:34:40.237 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.177.5, , ] 2021-09-02T07:34:40.238 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.177.5, , ] 2021-09-02T07:34:40.238 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:40.238+0000 7fd3d77b4380 0 bucket checkpoint complete 2021-09-02T07:34:40.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:40.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:40.253 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:40.302 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:40.302 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:40.324 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:40.326+0000 7fd6e32c5380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:40.325 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:40.326+0000 7fd6e32c5380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:40.325 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:40.326+0000 7fd6e32c5380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:34:40.325 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:40.326+0000 7fd6e32c5380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:40.362 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:40.363+0000 7fd6e32c5380 1 bucket sync caught up with source: 2021-09-02T07:34:40.362 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.177.5, , ] 2021-09-02T07:34:40.362 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.177.5, , ] 2021-09-02T07:34:40.362 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:40.363+0000 7fd6e32c5380 0 bucket checkpoint complete 2021-09-02T07:34:40.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:40.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:40.379 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:40.427 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:40.428 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:40.449 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:40.451+0000 7f466e6be380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:40.450 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:40.451+0000 7f466e6be380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:40.450 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:40.451+0000 7f466e6be380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:34:40.450 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:40.451+0000 7f466e6be380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:40.480 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:40.482+0000 7f466e6be380 1 bucket sync caught up with source: 2021-09-02T07:34:40.481 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.193.5, , ] 2021-09-02T07:34:40.481 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.193.5, , ] 2021-09-02T07:34:40.481 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:40.482+0000 7f466e6be380 0 bucket checkpoint complete 2021-09-02T07:34:40.496 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:40.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:40.497 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:40.548 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:40.549 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:40.570 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:40.572+0000 7fac10554380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:40.571 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:40.572+0000 7fac10554380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:40.571 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:40.572+0000 7fac10554380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:34:40.571 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:40.572+0000 7fac10554380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:40.601 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:40.603+0000 7fac10554380 1 bucket sync caught up with source: 2021-09-02T07:34:40.602 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000001.193.5, , ] 2021-09-02T07:34:40.602 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000001.193.5, , ] 2021-09-02T07:34:40.603 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:40.603+0000 7fac10554380 0 bucket checkpoint complete 2021-09-02T07:34:40.620 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-20 zones={test-zone2, test-zone1} 2021-09-02T07:34:40.626 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:34:40.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:40.672 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:34:40.676 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:40.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:40.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:40.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:40.846 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-20 zones={test-zone2, test-zone1} 2021-09-02T07:34:40.847 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-20 zones={test-zone3, test-zone1} 2021-09-02T07:34:40.852 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:34:40.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:40.900 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:34:40.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:40.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:40.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:41.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:41.078 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-20 zones={test-zone3, test-zone1} 2021-09-02T07:34:41.080 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-20 zones={test-zone3, test-zone2} 2021-09-02T07:34:41.086 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:34:41.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:41.136 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:34:41.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:41.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:41.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:41.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:41.321 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-20 zones={test-zone3, test-zone2} 2021-09-02T07:34:41.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:41.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:41.329 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:41.372 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:41.372 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:41.418 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T07:34:41.433 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:34:41.433 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:41.434 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:34:41.434 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:32:10.734248Z" 2021-09-02T07:34:41.434 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:41.434 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:41.434 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568081.330051_99.1", 2021-09-02T07:34:41.435 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:34:41.330051Z" 2021-09-02T07:34:41.435 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:41.435 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:41.436 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:41.436 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:34:41.436 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:41.436 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:41.437 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:41.437 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:34:41.437 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:41.437 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:34:41.438 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T07:34:41.439 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:41.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:41.440 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:41.483 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:41.483 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:41.563 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:34:41.563 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:34:41.564 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:34:41.564 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:34:41.564 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:34:41.564 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:34:41.565 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:41.565 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:41.565 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:34:41.566 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:41.566 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:34:41.566 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:41.567 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:41.567 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:34:41.567 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:41.567 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:34:41.568 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:41.568 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:34:41.568 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:41.568 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:41.569 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:41.569 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:41.569 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:34:41.569 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:41.569 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:41.570 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568081.326211_98.1", 2021-09-02T07:34:41.570 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:41.570 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:41.570 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:41.571 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:34:41.326211Z", 2021-09-02T07:34:41.571 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:41.571 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:41.571 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:41.572 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:41.572 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:34:41.572 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:41.572 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:41.572 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:41.573 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:41.573 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:41.573 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:41.573 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:34:41.574 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:41.574 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:41.574 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:41.574 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:41.574 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:34:41.575 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:41.575 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:41.575 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:41.575 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:41.576 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:41.576 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:41.576 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:34:41.576 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:41.576 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:41.577 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:41.577 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:34:41.578 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:41.578 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:34:41.578 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:34:41.578 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:34:41.579 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:41.579 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:34:41.580 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630568081.330051_99.1 target=1_1630568081.326211_98.1 2021-09-02T07:34:46.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:46.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:46.583 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:46.629 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:46.629 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:46.704 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:34:46.705 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:34:46.705 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:34:46.705 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:34:46.705 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:34:46.706 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:34:46.706 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:46.706 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:46.707 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:34:46.707 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:46.707 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:34:46.707 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:46.708 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:46.708 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:34:46.708 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:46.708 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:34:46.709 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:46.709 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:34:46.709 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:46.709 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:46.709 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:46.710 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:46.710 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:34:46.711 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:46.711 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:46.711 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568081.330051_99.1", 2021-09-02T07:34:46.711 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:46.712 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:46.712 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:46.712 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:34:41.330051Z", 2021-09-02T07:34:46.712 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:46.713 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:46.713 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:46.713 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:46.713 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:34:46.714 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:46.714 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:46.714 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:46.714 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:46.715 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:46.715 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:46.715 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:34:46.715 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:46.716 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:46.716 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:46.716 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:46.717 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:34:46.717 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:46.717 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:46.718 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:46.718 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:46.718 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:46.718 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:46.719 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:34:46.719 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:46.719 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:46.719 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:46.720 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:34:46.720 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:46.720 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:34:46.720 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:34:46.721 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:34:46.721 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:46.721 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:34:46.723 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T07:34:46.723 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T07:34:46.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:46.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:46.725 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:46.774 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:46.774 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:46.848 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:34:46.849 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:34:46.849 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:34:46.849 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:34:46.850 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:34:46.850 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:34:46.850 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:34:46.850 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:46.851 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:34:46.851 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:46.851 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:34:46.851 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:46.852 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:34:46.852 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:34:46.852 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:46.852 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:34:46.853 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:46.853 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:34:46.853 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:34:46.854 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:46.854 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:46.855 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:46.855 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:34:46.855 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:46.855 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:34:46.856 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568081.330051_99.1", 2021-09-02T07:34:46.856 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:46.856 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:46.856 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:46.857 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:34:41.330051Z", 2021-09-02T07:34:46.857 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:34:46.857 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:46.857 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:46.857 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:46.858 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:34:46.858 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:46.858 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:34:46.858 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:46.859 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:46.859 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:46.859 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:46.859 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:34:46.860 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:34:46.860 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:46.860 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:46.860 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:46.860 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:34:46.861 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:46.861 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:34:46.861 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:46.861 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:46.862 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:46.862 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:46.862 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:34:46.862 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:34:46.862 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:46.863 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:46.863 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:34:46.863 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:46.863 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:34:46.864 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:34:46.864 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:34:46.864 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:46.864 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:34:46.865 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T07:34:46.866 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T07:34:46.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:46.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:46.869 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:46.911 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:46.911 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:46.985 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:34:46.985 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:34:46.986 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:34:46.986 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:34:46.986 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:34:46.986 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:34:46.987 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:46.987 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:46.987 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:34:46.987 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:46.988 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:34:46.988 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:46.988 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:46.988 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:34:46.989 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:46.989 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:34:46.989 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:46.989 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:34:46.989 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:46.990 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:46.990 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:46.990 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:46.990 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:34:46.991 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:46.991 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:46.991 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568081.330051_99.1", 2021-09-02T07:34:46.991 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:46.992 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:46.992 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:46.992 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:34:41.330051Z", 2021-09-02T07:34:46.992 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:46.992 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:46.993 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:46.993 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:46.993 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:34:46.993 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:46.994 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:46.994 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:46.994 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:46.994 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:46.995 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:46.995 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:34:46.995 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:46.995 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:46.995 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:46.996 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:46.996 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:34:46.996 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:46.996 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:46.997 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:46.997 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:46.997 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:46.997 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:46.998 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:34:46.998 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:46.998 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:46.998 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:46.998 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:34:46.999 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:46.999 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:34:46.999 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:34:46.999 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:34:47.000 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:47.000 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:34:47.001 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-02T07:34:47.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:47.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:47.048 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:47.096 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:47.097 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:47.120 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:47.122+0000 7f60a32c6380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:47.120 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:47.122+0000 7f60a32c6380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:47.121 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:47.122+0000 7f60a32c6380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:34:47.121 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:47.122+0000 7f60a32c6380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:47.165 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:47.167+0000 7f60a32c6380 1 waiting for incremental sync to catch up: 2021-09-02T07:34:47.165 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000001.11.5, , ] 2021-09-02T07:34:47.166 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000003.14.12, , ] 2021-09-02T07:34:52.167 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:52.169+0000 7f60a32c6380 1 bucket sync caught up with source: 2021-09-02T07:34:52.167 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000003.14.12, , ] 2021-09-02T07:34:52.168 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000003.14.12, , ] 2021-09-02T07:34:52.168 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:52.169+0000 7f60a32c6380 0 bucket checkpoint complete 2021-09-02T07:34:52.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:52.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:52.189 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:52.245 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:52.245 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:52.271 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.273+0000 7f6c596da380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:52.271 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.273+0000 7f6c596da380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:52.272 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.273+0000 7f6c596da380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:34:52.272 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.273+0000 7f6c596da380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:52.304 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.306+0000 7f6c596da380 1 bucket sync caught up with source: 2021-09-02T07:34:52.304 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.14.12, , ] 2021-09-02T07:34:52.304 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.14.12, , ] 2021-09-02T07:34:52.305 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.306+0000 7f6c596da380 0 bucket checkpoint complete 2021-09-02T07:34:52.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:52.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:52.320 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:52.374 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:52.374 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:52.398 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:52.401+0000 7f7bb3354380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:52.399 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:52.401+0000 7f7bb3354380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:52.399 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:52.401+0000 7f7bb3354380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:34:52.400 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:52.401+0000 7f7bb3354380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:52.434 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:52.437+0000 7f7bb3354380 1 bucket sync caught up with source: 2021-09-02T07:34:52.435 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000003.14.12, , ] 2021-09-02T07:34:52.435 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000003.14.12, , ] 2021-09-02T07:34:52.435 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:52.437+0000 7f7bb3354380 0 bucket checkpoint complete 2021-09-02T07:34:52.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:52.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:52.451 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:52.512 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:52.513 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:52.535 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.537+0000 7f68307a7380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:52.536 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.537+0000 7f68307a7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:52.536 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.537+0000 7f68307a7380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:34:52.536 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.537+0000 7f68307a7380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:52.565 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.567+0000 7f68307a7380 1 bucket sync caught up with source: 2021-09-02T07:34:52.565 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000004.202.9, , ] 2021-09-02T07:34:52.565 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000004.202.9, , ] 2021-09-02T07:34:52.566 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.567+0000 7f68307a7380 0 bucket checkpoint complete 2021-09-02T07:34:52.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:52.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:52.581 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:52.630 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:52.631 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:52.652 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.654+0000 7f2d11089380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:52.652 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.654+0000 7f2d11089380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:52.652 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.654+0000 7f2d11089380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:34:52.653 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.654+0000 7f2d11089380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:52.683 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.685+0000 7f2d11089380 1 bucket sync caught up with source: 2021-09-02T07:34:52.683 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000004.202.9, , ] 2021-09-02T07:34:52.684 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000004.202.9, , ] 2021-09-02T07:34:52.684 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.685+0000 7f2d11089380 0 bucket checkpoint complete 2021-09-02T07:34:52.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:52.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:52.701 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:52.749 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:52.749 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:52.772 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.774+0000 7f3c0e9a7380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:52.772 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.774+0000 7f3c0e9a7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:52.773 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.774+0000 7f3c0e9a7380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:34:52.773 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.774+0000 7f3c0e9a7380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:52.804 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.805+0000 7f3c0e9a7380 1 bucket sync caught up with source: 2021-09-02T07:34:52.804 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000004.218.9, , ] 2021-09-02T07:34:52.804 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000004.218.9, , ] 2021-09-02T07:34:52.804 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:52.805+0000 7f3c0e9a7380 0 bucket checkpoint complete 2021-09-02T07:34:52.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:52.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:52.825 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:52.874 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:52.874 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:52.897 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:52.899+0000 7f392b163380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:52.898 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:52.899+0000 7f392b163380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:52.898 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:52.899+0000 7f392b163380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:34:52.898 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:52.899+0000 7f392b163380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:52.930 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:52.932+0000 7f392b163380 1 bucket sync caught up with source: 2021-09-02T07:34:52.930 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000004.218.9, , ] 2021-09-02T07:34:52.930 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000004.218.9, , ] 2021-09-02T07:34:52.931 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:52.932+0000 7f392b163380 0 bucket checkpoint complete 2021-09-02T07:34:52.951 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-20 zones={test-zone2, test-zone1} 2021-09-02T07:34:52.957 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:34:53.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:53.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:53.004 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:34:53.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:53.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:53.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:53.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:53.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:53.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:53.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:53.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:53.294 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-20 zones={test-zone2, test-zone1} 2021-09-02T07:34:53.294 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-20 zones={test-zone3, test-zone1} 2021-09-02T07:34:53.300 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:34:53.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:53.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:53.348 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:34:53.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:53.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:53.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:53.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:53.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:53.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:53.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:53.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:53.634 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-20 zones={test-zone3, test-zone1} 2021-09-02T07:34:53.634 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-20 zones={test-zone3, test-zone2} 2021-09-02T07:34:53.640 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:34:53.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:53.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:53.692 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:34:53.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:53.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:34:53.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:53.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:53.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:53.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:53.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:53.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:34:53.981 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-20 zones={test-zone3, test-zone2} 2021-09-02T07:34:53.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:53.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:53.990 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:54.031 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:54.032 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:54.076 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T07:34:54.092 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:34:54.092 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:54.092 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:34:54.093 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:32:10.734248Z" 2021-09-02T07:34:54.093 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:54.093 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:54.093 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568093.990716_101.1", 2021-09-02T07:34:54.094 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:34:53.990716Z" 2021-09-02T07:34:54.094 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:54.094 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:54.094 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:54.095 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:34:54.095 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:54.095 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:54.095 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:54.096 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:34:54.096 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:54.096 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:34:54.097 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T07:34:54.098 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:54.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:54.099 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:54.144 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:54.144 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:54.215 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:34:54.216 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:34:54.216 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:34:54.216 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:34:54.217 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:34:54.217 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:34:54.217 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:54.217 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:54.218 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:34:54.218 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:54.218 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:34:54.218 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:54.219 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:54.219 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:34:54.219 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:54.219 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:34:54.219 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:54.220 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:34:54.220 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:54.220 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:54.220 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:54.221 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:54.221 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:34:54.221 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:54.221 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:54.221 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568093.986709_100.1", 2021-09-02T07:34:54.222 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:54.222 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:54.222 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:54.222 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:34:53.986709Z", 2021-09-02T07:34:54.223 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:54.223 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:54.223 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:54.223 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:54.223 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:34:54.224 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:54.224 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:54.224 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:54.224 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:54.225 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:54.225 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:54.225 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:34:54.225 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:54.225 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:54.226 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:54.226 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:54.226 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:34:54.226 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:54.227 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:54.227 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:54.227 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:54.227 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:54.228 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:54.228 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:34:54.228 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:54.228 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:54.228 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:54.229 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:34:54.229 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:54.229 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:34:54.229 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:34:54.229 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:34:54.230 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:54.230 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:34:54.231 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630568093.990716_101.1 target=1_1630568093.986709_100.1 2021-09-02T07:34:59.232 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:59.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:59.234 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:59.282 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:59.282 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:59.364 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:34:59.364 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:34:59.364 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:34:59.365 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:34:59.365 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:34:59.365 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:34:59.365 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:59.366 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:59.366 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:34:59.366 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:59.366 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:34:59.366 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:59.367 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:59.367 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:34:59.367 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:59.367 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:34:59.368 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:59.368 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:34:59.368 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:59.369 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:59.369 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:59.369 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:59.369 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:34:59.370 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:59.370 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:59.370 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568093.990716_101.1", 2021-09-02T07:34:59.370 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:59.371 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:59.371 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:59.371 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:34:53.990716Z", 2021-09-02T07:34:59.371 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:59.372 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:59.372 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:59.372 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:59.372 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:34:59.372 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:59.373 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:59.373 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:59.373 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:59.373 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:59.374 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:59.374 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:34:59.374 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:59.374 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:59.374 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:59.375 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:59.375 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:34:59.375 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:59.375 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:59.376 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:59.376 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:59.376 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:59.376 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:59.376 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:34:59.377 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:59.377 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:59.377 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:59.377 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:34:59.378 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:59.378 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:34:59.378 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:34:59.378 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:34:59.378 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:59.379 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:34:59.380 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T07:34:59.380 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T07:34:59.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:59.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:59.382 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:59.432 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:59.433 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:59.506 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:34:59.507 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:34:59.507 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:34:59.507 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:34:59.508 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:34:59.508 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:34:59.509 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:34:59.509 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:59.510 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:34:59.510 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:59.511 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:34:59.511 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:59.511 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:34:59.512 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:34:59.512 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:59.513 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:34:59.513 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:59.514 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:34:59.514 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:34:59.514 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:59.515 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:59.515 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:59.516 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:34:59.516 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:59.517 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:34:59.517 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568093.990716_101.1", 2021-09-02T07:34:59.517 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:59.518 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:59.518 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:59.519 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:34:53.990716Z", 2021-09-02T07:34:59.519 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:34:59.519 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:59.520 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:59.520 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:59.521 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:34:59.521 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:59.521 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:34:59.522 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:59.522 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:59.523 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:59.523 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:59.523 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:34:59.524 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:34:59.524 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:59.525 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:59.525 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:34:59.526 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:34:59.526 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:34:59.527 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:34:59.527 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:34:59.528 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:34:59.528 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:34:59.529 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:34:59.529 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:34:59.529 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:34:59.530 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:59.531 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:59.532 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:34:59.532 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:34:59.533 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:34:59.533 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:34:59.534 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:34:59.534 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:34:59.535 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:34:59.537 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T07:34:59.538 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T07:34:59.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:59.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:34:59.541 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:34:59.588 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:59.588 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:59.652 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:34:59.653 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:34:59.654 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:34:59.655 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:34:59.655 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:34:59.655 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:34:59.655 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:59.656 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:59.656 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:34:59.656 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:59.656 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:34:59.656 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:59.657 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:59.657 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:34:59.657 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:59.657 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:34:59.658 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:59.658 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:34:59.658 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:59.659 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:59.659 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:59.659 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:59.659 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:34:59.659 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:59.660 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:59.660 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568093.990716_101.1", 2021-09-02T07:34:59.660 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:59.660 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:59.661 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:59.661 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:34:53.990716Z", 2021-09-02T07:34:59.661 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:59.661 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:59.662 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:59.662 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:59.662 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:34:59.662 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:59.663 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:59.663 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:59.663 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:59.663 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:59.663 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:59.664 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:34:59.664 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:59.664 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:59.664 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:59.664 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:34:59.665 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:34:59.665 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:34:59.665 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:34:59.665 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:34:59.666 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:34:59.666 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:34:59.666 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:34:59.666 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:34:59.667 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:34:59.667 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:59.667 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:59.667 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:34:59.668 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:34:59.668 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:34:59.668 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:34:59.668 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:34:59.668 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:34:59.669 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:34:59.670 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-02T07:34:59.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:59.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:59.721 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:59.771 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:59.772 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:59.793 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:59.796+0000 7fc4ea3ba380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:59.794 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:59.796+0000 7fc4ea3ba380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:59.794 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:59.796+0000 7fc4ea3ba380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:34:59.794 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:59.796+0000 7fc4ea3ba380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:59.828 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:59.830+0000 7fc4ea3ba380 1 bucket sync caught up with source: 2021-09-02T07:34:59.828 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000005.18.17, , ] 2021-09-02T07:34:59.828 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000005.18.17, , ] 2021-09-02T07:34:59.829 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:34:59.830+0000 7fc4ea3ba380 0 bucket checkpoint complete 2021-09-02T07:34:59.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:59.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:59.847 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:34:59.897 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:34:59.898 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:34:59.921 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:59.922+0000 7f3aaa277380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:34:59.921 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:59.922+0000 7f3aaa277380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:34:59.921 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:59.922+0000 7f3aaa277380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:34:59.922 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:59.922+0000 7f3aaa277380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:34:59.954 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:59.956+0000 7f3aaa277380 1 bucket sync caught up with source: 2021-09-02T07:34:59.955 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000005.18.17, , ] 2021-09-02T07:34:59.955 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000005.18.17, , ] 2021-09-02T07:34:59.955 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:34:59.956+0000 7f3aaa277380 0 bucket checkpoint complete 2021-09-02T07:34:59.971 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:59.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:34:59.972 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:00.022 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:00.022 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:00.046 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:00.048+0000 7ff81970f380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:00.046 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:00.048+0000 7ff81970f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:00.047 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:00.048+0000 7ff81970f380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:35:00.047 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:00.048+0000 7ff81970f380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:00.092 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:00.094+0000 7ff81970f380 1 bucket sync caught up with source: 2021-09-02T07:35:00.093 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000005.18.17, , ] 2021-09-02T07:35:00.093 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000005.18.17, , ] 2021-09-02T07:35:00.093 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:00.094+0000 7ff81970f380 0 bucket checkpoint complete 2021-09-02T07:35:00.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:00.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:00.111 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:00.160 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:00.161 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:00.183 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:00.185+0000 7f06a426a380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:00.184 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:00.185+0000 7f06a426a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:00.184 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:00.185+0000 7f06a426a380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:35:00.184 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:00.185+0000 7f06a426a380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:00.214 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:00.216+0000 7f06a426a380 1 bucket sync caught up with source: 2021-09-02T07:35:00.214 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000007.228.14, , ] 2021-09-02T07:35:00.215 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000007.228.14, , ] 2021-09-02T07:35:00.215 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:00.216+0000 7f06a426a380 0 bucket checkpoint complete 2021-09-02T07:35:00.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:00.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:00.230 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:00.279 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:00.280 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:00.303 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:00.304+0000 7ff50f092380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:00.303 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:00.304+0000 7ff50f092380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:00.303 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:00.304+0000 7ff50f092380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:35:00.304 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:00.304+0000 7ff50f092380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:00.334 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:00.336+0000 7ff50f092380 1 bucket sync caught up with source: 2021-09-02T07:35:00.334 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000007.228.14, , ] 2021-09-02T07:35:00.334 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000007.228.14, , ] 2021-09-02T07:35:00.335 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:00.336+0000 7ff50f092380 0 bucket checkpoint complete 2021-09-02T07:35:00.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:00.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:00.353 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:00.401 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:00.401 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:00.423 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:00.424+0000 7ff667ecf380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:00.423 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:00.424+0000 7ff667ecf380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:00.423 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:00.424+0000 7ff667ecf380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:35:00.424 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:00.424+0000 7ff667ecf380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:00.454 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:00.455+0000 7ff667ecf380 1 bucket sync caught up with source: 2021-09-02T07:35:00.455 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000007.244.14, , ] 2021-09-02T07:35:00.455 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000007.244.14, , ] 2021-09-02T07:35:00.455 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:00.455+0000 7ff667ecf380 0 bucket checkpoint complete 2021-09-02T07:35:00.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:00.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:00.472 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:00.523 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:00.524 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:00.547 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:00.548+0000 7fe8586e3380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:00.547 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:00.548+0000 7fe8586e3380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:00.547 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:00.548+0000 7fe8586e3380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:35:00.547 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:00.548+0000 7fe8586e3380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:00.589 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:00.591+0000 7fe8586e3380 1 bucket sync caught up with source: 2021-09-02T07:35:00.589 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000007.244.14, , ] 2021-09-02T07:35:00.590 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000007.244.14, , ] 2021-09-02T07:35:00.590 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:00.591+0000 7fe8586e3380 0 bucket checkpoint complete 2021-09-02T07:35:00.604 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-20 zones={test-zone2, test-zone1} 2021-09-02T07:35:00.610 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:00.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:00.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:00.656 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:00.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:00.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:00.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:00.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:00.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:00.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:00.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:00.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:00.950 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-20 zones={test-zone2, test-zone1} 2021-09-02T07:35:00.951 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-20 zones={test-zone3, test-zone1} 2021-09-02T07:35:00.957 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:01.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:01.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:01.008 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:01.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:01.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:01.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:01.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:01.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:01.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:01.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:01.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:01.310 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-20 zones={test-zone3, test-zone1} 2021-09-02T07:35:01.310 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-20 zones={test-zone3, test-zone2} 2021-09-02T07:35:01.316 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:01.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:01.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:01.368 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:01.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:01.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:01.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:01.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:01.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:01.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:01.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:01.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:01.666 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-20 zones={test-zone3, test-zone2} 2021-09-02T07:35:01.667 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2021-09-02T07:35:01.668 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-21 2021-09-02T07:35:01.686 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=efybpk-22 2021-09-02T07:35:01.721 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=efybpk-23 2021-09-02T07:35:01.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:01.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:01.797 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:35:01.840 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:01.840 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:01.886 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T07:35:01.901 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:35:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:35:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:32:10.734248Z" 2021-09-02T07:35:01.902 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568101.795076_120.1", 2021-09-02T07:35:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:35:01.795076Z" 2021-09-02T07:35:01.903 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:35:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:35:01.904 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:35:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:35:01.905 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:01.906 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:35:01.907 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T07:35:01.907 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:01.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:01.908 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:35:01.953 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:01.954 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:02.027 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:35:02.028 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:35:02.028 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:35:02.028 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:35:02.028 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:35:02.028 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:35:02.029 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:02.029 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:02.029 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:35:02.029 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:02.030 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:35:02.030 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:02.030 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:02.030 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:35:02.031 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:02.031 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:35:02.031 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:02.031 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:35:02.031 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:02.032 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:02.032 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:02.032 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:02.032 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:35:02.033 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:02.033 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:02.033 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568101.678576_103.1", 2021-09-02T07:35:02.034 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:02.034 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:02.034 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:02.034 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:01.678576Z", 2021-09-02T07:35:02.035 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:02.035 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:02.035 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:02.035 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:02.036 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:35:02.036 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:02.036 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:02.037 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:02.037 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:02.037 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:02.037 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:02.038 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:02.038 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:02.038 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:02.038 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:02.038 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:02.039 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:35:02.039 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:02.039 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:02.039 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:02.040 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:02.040 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:02.040 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:02.041 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:02.041 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:02.041 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:02.041 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:02.041 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:35:02.042 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:02.042 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:35:02.042 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:35:02.042 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:35:02.043 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:02.043 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:35:02.044 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630568101.795076_120.1 target=1_1630568101.678576_103.1 2021-09-02T07:35:07.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:07.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:07.048 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:35:07.096 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:07.096 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:07.164 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:35:07.165 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:35:07.165 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:35:07.165 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:35:07.165 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:35:07.166 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:35:07.166 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:07.167 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:07.167 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:35:07.167 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:07.167 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:35:07.168 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:07.168 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:07.168 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:35:07.168 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:07.168 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:35:07.169 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:07.170 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:35:07.170 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:07.171 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:07.171 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:07.171 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:07.171 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:35:07.172 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:07.172 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:07.172 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568101.795076_120.1", 2021-09-02T07:35:07.172 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:07.172 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:07.173 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:07.173 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:01.795076Z", 2021-09-02T07:35:07.173 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:07.174 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:07.174 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:07.174 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:07.174 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:35:07.175 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:07.175 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:07.175 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:07.175 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:07.175 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:07.176 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:07.176 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:07.177 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:07.177 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:07.177 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:07.178 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:07.178 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:35:07.178 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:07.178 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:07.178 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:07.179 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:07.179 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:07.179 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:07.179 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:07.180 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:07.180 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:07.180 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:07.181 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:35:07.181 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:07.181 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:35:07.181 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:35:07.182 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:35:07.182 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:07.182 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:35:07.183 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T07:35:07.184 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T07:35:07.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:07.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:07.188 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:35:07.238 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:07.238 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:07.312 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:35:07.312 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:35:07.312 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:35:07.313 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:35:07.313 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:35:07.313 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:35:07.313 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:35:07.314 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:07.314 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:35:07.314 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:07.314 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:35:07.315 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:35:07.315 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:35:07.315 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:35:07.315 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:35:07.316 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:35:07.316 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:35:07.316 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:35:07.317 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:35:07.317 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:07.317 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:07.317 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:07.318 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:35:07.318 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:35:07.318 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:35:07.318 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568101.795076_120.1", 2021-09-02T07:35:07.319 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:35:07.319 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:35:07.319 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:35:07.319 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:01.795076Z", 2021-09-02T07:35:07.319 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:35:07.320 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:07.320 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:07.320 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:07.320 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:35:07.321 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:35:07.321 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:35:07.321 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:35:07.322 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:35:07.322 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:35:07.323 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:35:07.323 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:35:07.323 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:35:07.324 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:07.324 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:07.324 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:07.324 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:35:07.325 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:35:07.325 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:35:07.325 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:35:07.325 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:35:07.326 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:35:07.326 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:35:07.326 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:35:07.326 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:35:07.327 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:07.327 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:07.327 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:35:07.327 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:07.328 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:35:07.328 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:35:07.328 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:35:07.328 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:07.328 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:35:07.330 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T07:35:07.330 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T07:35:07.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:07.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:07.332 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:35:07.376 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:07.377 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:07.448 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:35:07.449 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:35:07.449 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:35:07.449 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:35:07.450 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:35:07.450 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:35:07.450 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:07.450 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:07.451 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:35:07.451 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:07.451 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:35:07.451 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:07.452 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:07.452 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:35:07.452 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:07.452 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:35:07.453 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:07.453 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:35:07.453 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:07.453 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:07.453 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:07.454 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:07.454 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:35:07.454 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:07.455 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:07.455 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568101.795076_120.1", 2021-09-02T07:35:07.455 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:07.455 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:07.456 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:07.456 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:01.795076Z", 2021-09-02T07:35:07.456 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:07.456 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:07.456 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:07.457 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:07.457 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:35:07.457 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:07.457 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:07.458 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:07.458 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:07.458 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:07.458 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:07.458 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:07.459 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:07.459 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:07.459 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:07.459 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:07.460 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:35:07.460 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:07.460 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:07.460 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:07.461 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:07.461 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:07.461 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:07.461 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:07.461 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:07.462 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:07.462 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:07.462 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:35:07.462 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:07.463 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:35:07.463 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:35:07.463 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:35:07.463 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:07.464 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:35:07.465 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-02T07:35:07.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:07.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:07.647 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:07.702 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:07.702 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:07.725 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:07.726+0000 7fd6fce63380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:07.725 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:07.726+0000 7fd6fce63380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:07.725 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:07.726+0000 7fd6fce63380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:35:07.726 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:07.726+0000 7fd6fce63380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:07.760 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:07.761+0000 7fd6fce63380 1 bucket sync caught up with source: 2021-09-02T07:35:07.760 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000003.9.11, , ] 2021-09-02T07:35:07.760 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000003.9.11, , ] 2021-09-02T07:35:07.761 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:07.761+0000 7fd6fce63380 0 bucket checkpoint complete 2021-09-02T07:35:07.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:07.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:07.790 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:07.845 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:07.845 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:07.873 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:07.874+0000 7f84c339e380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:07.873 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:07.874+0000 7f84c339e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:07.873 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:07.874+0000 7f84c339e380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:35:07.873 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:07.874+0000 7f84c339e380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:07.911 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:07.912+0000 7f84c339e380 1 bucket sync caught up with source: 2021-09-02T07:35:07.911 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.9.11, , ] 2021-09-02T07:35:07.911 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.9.11, , ] 2021-09-02T07:35:07.912 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:07.912+0000 7f84c339e380 0 bucket checkpoint complete 2021-09-02T07:35:07.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:07.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:07.931 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:07.984 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:07.985 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:08.006 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:08.007+0000 7f2ee6679380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:08.006 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:08.007+0000 7f2ee6679380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:08.006 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:08.007+0000 7f2ee6679380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:35:08.007 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:08.007+0000 7f2ee6679380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:08.047 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:08.048+0000 7f2ee6679380 1 bucket sync caught up with source: 2021-09-02T07:35:08.047 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000003.9.11, , ] 2021-09-02T07:35:08.047 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000003.9.11, , ] 2021-09-02T07:35:08.048 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:08.048+0000 7f2ee6679380 0 bucket checkpoint complete 2021-09-02T07:35:08.065 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:08.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:08.066 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:08.116 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:08.116 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:08.138 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:08.140+0000 7f5c8bfd9380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:08.138 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:08.140+0000 7f5c8bfd9380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:08.139 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:08.140+0000 7f5c8bfd9380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:35:08.139 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:08.140+0000 7f5c8bfd9380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:08.178 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:08.179+0000 7f5c8bfd9380 1 bucket sync caught up with source: 2021-09-02T07:35:08.178 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000002.144.5, , ] 2021-09-02T07:35:08.178 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000002.144.5, , ] 2021-09-02T07:35:08.179 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:08.179+0000 7f5c8bfd9380 0 bucket checkpoint complete 2021-09-02T07:35:08.195 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:08.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:08.196 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:08.245 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:08.245 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:08.267 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:08.269+0000 7f131c738380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:08.267 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:08.269+0000 7f131c738380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:08.267 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:08.269+0000 7f131c738380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:35:08.268 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:08.269+0000 7f131c738380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:08.309 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:08.311+0000 7f131c738380 1 bucket sync caught up with source: 2021-09-02T07:35:08.309 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000002.144.5, , ] 2021-09-02T07:35:08.310 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000002.144.5, , ] 2021-09-02T07:35:08.310 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:08.311+0000 7f131c738380 0 bucket checkpoint complete 2021-09-02T07:35:08.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:08.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:08.330 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:08.378 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:08.379 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:08.401 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:08.403+0000 7f73a583a380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:08.401 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:08.403+0000 7f73a583a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:08.401 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:08.403+0000 7f73a583a380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:35:08.402 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:08.403+0000 7f73a583a380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:08.434 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:08.436+0000 7f73a583a380 1 bucket sync caught up with source: 2021-09-02T07:35:08.434 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000002.184.5, , ] 2021-09-02T07:35:08.435 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000002.184.5, , ] 2021-09-02T07:35:08.435 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:08.436+0000 7f73a583a380 0 bucket checkpoint complete 2021-09-02T07:35:08.450 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:08.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:08.451 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:08.501 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:08.501 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:08.526 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:08.527+0000 7f581215b380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:08.527 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:08.527+0000 7f581215b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:08.527 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:08.527+0000 7f581215b380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:35:08.527 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:08.527+0000 7f581215b380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:08.569 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:08.570+0000 7f581215b380 1 bucket sync caught up with source: 2021-09-02T07:35:08.569 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000002.184.5, , ] 2021-09-02T07:35:08.569 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000002.184.5, , ] 2021-09-02T07:35:08.569 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:08.570+0000 7f581215b380 0 bucket checkpoint complete 2021-09-02T07:35:08.586 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-21 zones={test-zone2, test-zone1} 2021-09-02T07:35:08.592 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:08.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:08.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:08.644 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:08.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:08.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:08.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:08.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:08.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:08.774 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-21 zones={test-zone2, test-zone1} 2021-09-02T07:35:08.774 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-21 zones={test-zone3, test-zone1} 2021-09-02T07:35:08.780 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:08.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:08.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:08.828 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:08.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:08.832 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:08.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:08.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:08.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:08.958 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-21 zones={test-zone3, test-zone1} 2021-09-02T07:35:08.959 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-21 zones={test-zone3, test-zone2} 2021-09-02T07:35:08.965 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:09.011 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:09.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:09.012 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:09.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:09.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:09.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:09.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:09.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:09.135 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-21 zones={test-zone3, test-zone2} 2021-09-02T07:35:09.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:09.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:09.137 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:09.187 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:09.187 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:09.209 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:09.210+0000 7fc2a1bf4380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:09.209 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:09.210+0000 7fc2a1bf4380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:09.210 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:09.210+0000 7fc2a1bf4380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:35:09.210 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:09.210+0000 7fc2a1bf4380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:09.252 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:09.253+0000 7fc2a1bf4380 1 bucket sync caught up with source: 2021-09-02T07:35:09.253 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000002.15.5, , ] 2021-09-02T07:35:09.253 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000002.15.5, , ] 2021-09-02T07:35:09.253 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:09.253+0000 7fc2a1bf4380 0 bucket checkpoint complete 2021-09-02T07:35:09.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:09.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:09.270 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:09.319 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:09.320 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:09.342 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.344+0000 7f00a04b7380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:09.343 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.344+0000 7f00a04b7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:09.343 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.344+0000 7f00a04b7380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:35:09.343 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.344+0000 7f00a04b7380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:09.380 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.381+0000 7f00a04b7380 1 bucket sync caught up with source: 2021-09-02T07:35:09.381 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000002.15.5, , ] 2021-09-02T07:35:09.381 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000002.15.5, , ] 2021-09-02T07:35:09.382 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.381+0000 7f00a04b7380 0 bucket checkpoint complete 2021-09-02T07:35:09.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:09.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:09.400 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:09.451 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:09.451 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:09.473 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:09.473+0000 7f9dc7bc9380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:09.473 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:09.473+0000 7f9dc7bc9380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:09.474 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:09.473+0000 7f9dc7bc9380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:35:09.474 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:09.473+0000 7f9dc7bc9380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:09.516 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:09.517+0000 7f9dc7bc9380 1 bucket sync caught up with source: 2021-09-02T07:35:09.516 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000002.15.5, , ] 2021-09-02T07:35:09.517 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000002.15.5, , ] 2021-09-02T07:35:09.517 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:09.517+0000 7f9dc7bc9380 0 bucket checkpoint complete 2021-09-02T07:35:09.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:09.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:09.535 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:09.585 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:09.586 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:09.607 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.608+0000 7f4fbdb43380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:09.607 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.608+0000 7f4fbdb43380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:09.607 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.608+0000 7f4fbdb43380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:35:09.608 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.608+0000 7f4fbdb43380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:09.651 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.653+0000 7f4fbdb43380 1 bucket sync caught up with source: 2021-09-02T07:35:09.652 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.250.11, , ] 2021-09-02T07:35:09.652 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.250.11, , ] 2021-09-02T07:35:09.652 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.653+0000 7f4fbdb43380 0 bucket checkpoint complete 2021-09-02T07:35:09.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:09.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:09.668 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:09.716 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:09.716 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:09.737 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.739+0000 7f1109a4d380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:09.738 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.739+0000 7f1109a4d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:09.738 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.739+0000 7f1109a4d380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:35:09.738 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.739+0000 7f1109a4d380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:09.780 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.781+0000 7f1109a4d380 1 bucket sync caught up with source: 2021-09-02T07:35:09.780 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.250.11, , ] 2021-09-02T07:35:09.780 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.250.11, , ] 2021-09-02T07:35:09.781 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.781+0000 7f1109a4d380 0 bucket checkpoint complete 2021-09-02T07:35:09.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:09.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:09.797 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:09.847 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:09.848 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:09.871 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.872+0000 7fa6b826d380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:09.871 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.872+0000 7fa6b826d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:09.871 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.872+0000 7fa6b826d380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:35:09.872 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.872+0000 7fa6b826d380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:09.901 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.903+0000 7fa6b826d380 1 bucket sync caught up with source: 2021-09-02T07:35:09.902 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000002.221.5, , ] 2021-09-02T07:35:09.902 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000002.221.5, , ] 2021-09-02T07:35:09.902 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:09.903+0000 7fa6b826d380 0 bucket checkpoint complete 2021-09-02T07:35:09.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:09.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:09.918 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:09.967 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:09.967 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:09.989 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:09.991+0000 7fc4c69a4380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:09.990 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:09.991+0000 7fc4c69a4380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:09.990 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:09.991+0000 7fc4c69a4380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:35:09.990 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:09.991+0000 7fc4c69a4380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:10.027 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:10.028+0000 7fc4c69a4380 1 bucket sync caught up with source: 2021-09-02T07:35:10.027 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000002.221.5, , ] 2021-09-02T07:35:10.028 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000002.221.5, , ] 2021-09-02T07:35:10.028 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:10.028+0000 7fc4c69a4380 0 bucket checkpoint complete 2021-09-02T07:35:10.043 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-22 zones={test-zone2, test-zone1} 2021-09-02T07:35:10.049 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:10.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:10.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:10.100 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:10.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:10.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:10.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:10.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:10.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:10.234 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-22 zones={test-zone2, test-zone1} 2021-09-02T07:35:10.235 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-22 zones={test-zone3, test-zone1} 2021-09-02T07:35:10.241 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:10.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:10.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:10.289 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:10.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:10.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:10.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:10.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:10.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:10.420 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-22 zones={test-zone3, test-zone1} 2021-09-02T07:35:10.421 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-22 zones={test-zone3, test-zone2} 2021-09-02T07:35:10.427 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:10.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:10.477 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:10.477 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:10.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:10.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:10.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:10.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:10.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:10.601 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-22 zones={test-zone3, test-zone2} 2021-09-02T07:35:10.602 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:10.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:10.603 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:10.652 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:10.652 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:10.674 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:10.675+0000 7f9459ded380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:10.674 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:10.675+0000 7f9459ded380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:10.674 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:10.675+0000 7f9459ded380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:35:10.675 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:10.675+0000 7f9459ded380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:10.709 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:10.710+0000 7f9459ded380 1 bucket sync caught up with source: 2021-09-02T07:35:10.709 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000002.9.5, , ] 2021-09-02T07:35:10.710 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000002.9.5, , ] 2021-09-02T07:35:10.710 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:10.710+0000 7f9459ded380 0 bucket checkpoint complete 2021-09-02T07:35:10.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:10.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:10.728 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:10.776 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:10.777 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:10.799 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:10.801+0000 7fa3a42aa380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:10.800 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:10.801+0000 7fa3a42aa380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:10.800 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:10.801+0000 7fa3a42aa380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:35:10.800 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:10.801+0000 7fa3a42aa380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:10.845 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:10.846+0000 7fa3a42aa380 1 bucket sync caught up with source: 2021-09-02T07:35:10.846 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000002.9.5, , ] 2021-09-02T07:35:10.846 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000002.9.5, , ] 2021-09-02T07:35:10.846 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:10.846+0000 7fa3a42aa380 0 bucket checkpoint complete 2021-09-02T07:35:10.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:10.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:10.863 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:10.913 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:10.913 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:10.935 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:10.936+0000 7f26b7484380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:10.936 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:10.936+0000 7f26b7484380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:10.936 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:10.936+0000 7f26b7484380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:35:10.936 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:10.936+0000 7f26b7484380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:10.979 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:10.980+0000 7f26b7484380 1 bucket sync caught up with source: 2021-09-02T07:35:10.979 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000002.9.5, , ] 2021-09-02T07:35:10.979 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000002.9.5, , ] 2021-09-02T07:35:10.980 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:10.980+0000 7f26b7484380 0 bucket checkpoint complete 2021-09-02T07:35:10.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:10.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:10.999 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:11.048 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:11.049 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:11.073 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:11.074+0000 7fb8b7a4e380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:11.074 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:11.074+0000 7fb8b7a4e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:11.074 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:11.074+0000 7fb8b7a4e380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:35:11.075 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:11.074+0000 7fb8b7a4e380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:11.104 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:11.106+0000 7fb8b7a4e380 1 bucket sync caught up with source: 2021-09-02T07:35:11.105 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000002.133.5, , ] 2021-09-02T07:35:11.105 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000002.133.5, , ] 2021-09-02T07:35:11.105 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:11.106+0000 7fb8b7a4e380 0 bucket checkpoint complete 2021-09-02T07:35:11.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:11.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:11.121 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:11.170 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:11.170 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:11.192 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:11.194+0000 7f37db647380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:11.192 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:11.194+0000 7f37db647380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:11.193 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:11.194+0000 7f37db647380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:35:11.193 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:11.194+0000 7f37db647380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:11.222 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:11.224+0000 7f37db647380 1 bucket sync caught up with source: 2021-09-02T07:35:11.222 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000002.133.5, , ] 2021-09-02T07:35:11.223 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000002.133.5, , ] 2021-09-02T07:35:11.223 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:11.224+0000 7f37db647380 0 bucket checkpoint complete 2021-09-02T07:35:11.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:11.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:11.239 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:11.287 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:11.288 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:11.310 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:11.311+0000 7f4b71e6e380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:11.310 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:11.311+0000 7f4b71e6e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:11.310 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:11.311+0000 7f4b71e6e380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:35:11.310 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:11.311+0000 7f4b71e6e380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:11.340 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:11.341+0000 7f4b71e6e380 1 bucket sync caught up with source: 2021-09-02T07:35:11.340 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.178.11, , ] 2021-09-02T07:35:11.340 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.178.11, , ] 2021-09-02T07:35:11.340 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:11.341+0000 7f4b71e6e380 0 bucket checkpoint complete 2021-09-02T07:35:11.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:11.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:11.356 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:11.406 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:11.406 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:11.430 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:11.430+0000 7f9b6a357380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:11.431 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:11.430+0000 7f9b6a357380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:11.431 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:11.430+0000 7f9b6a357380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:35:11.431 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:11.430+0000 7f9b6a357380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:11.471 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:11.472+0000 7f9b6a357380 1 bucket sync caught up with source: 2021-09-02T07:35:11.472 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000003.178.11, , ] 2021-09-02T07:35:11.472 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000003.178.11, , ] 2021-09-02T07:35:11.472 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:11.472+0000 7f9b6a357380 0 bucket checkpoint complete 2021-09-02T07:35:11.487 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-23 zones={test-zone2, test-zone1} 2021-09-02T07:35:11.494 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:11.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:11.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:11.540 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:11.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:11.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:11.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:11.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:11.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:11.661 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-23 zones={test-zone2, test-zone1} 2021-09-02T07:35:11.662 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-23 zones={test-zone3, test-zone1} 2021-09-02T07:35:11.667 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:11.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:11.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:11.716 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:11.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:11.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:11.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:11.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:11.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:11.848 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-23 zones={test-zone3, test-zone1} 2021-09-02T07:35:11.849 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-23 zones={test-zone3, test-zone2} 2021-09-02T07:35:11.854 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:11.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:11.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:11.900 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:11.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:11.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:11.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:11.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:11.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:12.030 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-23 zones={test-zone3, test-zone2} 2021-09-02T07:35:12.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2021-09-02T07:35:12.032 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-24 2021-09-02T07:35:12.049 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=efybpk-25 2021-09-02T07:35:12.084 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=efybpk-26 2021-09-02T07:35:12.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:12.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:12.171 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:35:12.217 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:12.217 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:12.265 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T07:35:12.282 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:35:12.282 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:12.282 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:35:12.283 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:32:10.734248Z" 2021-09-02T07:35:12.283 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:12.283 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:12.283 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568112.168881_147.1", 2021-09-02T07:35:12.283 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:35:12.168881Z" 2021-09-02T07:35:12.284 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:12.284 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:12.284 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:35:12.284 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:35:12.285 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:12.285 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:12.285 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:35:12.285 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:35:12.286 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:12.286 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:35:12.287 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T07:35:12.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:12.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:12.289 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:35:12.331 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:12.331 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:12.398 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:35:12.399 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:35:12.399 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:35:12.399 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:35:12.400 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:35:12.400 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:35:12.400 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:12.400 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:12.401 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:35:12.401 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:12.401 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:35:12.402 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:12.403 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:12.403 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:35:12.404 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:12.404 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:35:12.404 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:12.404 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:35:12.405 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:12.405 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:12.405 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:12.405 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:12.406 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:35:12.406 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:12.406 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:12.406 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568112.042754_124.1", 2021-09-02T07:35:12.407 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:12.407 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:12.407 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:12.407 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:12.042754Z", 2021-09-02T07:35:12.408 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:12.408 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:12.408 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:12.408 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:12.408 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:35:12.409 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:12.409 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:12.409 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:12.409 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:12.410 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:12.410 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:12.410 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:12.410 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:12.410 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:12.411 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:12.411 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:12.411 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:35:12.411 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:12.412 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:12.412 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:12.412 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:12.413 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:12.413 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:12.413 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:12.413 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:12.414 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:12.414 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:12.414 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:35:12.415 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:12.415 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:35:12.415 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:35:12.415 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:35:12.416 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:12.416 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:35:12.417 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630568112.168881_147.1 target=1_1630568112.042754_124.1 2021-09-02T07:35:17.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:17.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:17.419 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:35:17.468 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:17.468 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:17.535 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:35:17.536 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:35:17.536 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:35:17.536 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:35:17.537 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:35:17.537 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:35:17.537 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:17.537 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:17.537 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:35:17.538 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:17.538 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:35:17.538 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:17.538 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:17.539 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:35:17.539 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:17.539 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:35:17.539 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:17.540 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:35:17.540 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:17.540 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:17.540 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:17.540 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:17.541 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:35:17.541 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:17.541 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:17.541 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568112.168881_147.1", 2021-09-02T07:35:17.542 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:17.542 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:17.542 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:17.542 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:12.168881Z", 2021-09-02T07:35:17.543 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:17.543 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:17.543 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:17.543 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:17.543 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:35:17.544 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:17.544 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:17.544 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:17.545 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:17.545 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:17.545 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:17.546 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:17.546 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:17.546 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:17.546 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:17.547 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:17.547 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:35:17.547 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:17.547 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:17.547 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:17.548 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:17.548 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:17.548 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:17.548 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:17.549 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:17.549 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:17.549 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:17.549 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:35:17.549 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:17.550 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:35:17.550 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:35:17.550 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:35:17.550 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:17.551 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:35:17.552 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T07:35:17.552 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T07:35:17.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:17.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:17.554 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:35:17.603 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:17.603 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:17.686 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:35:17.686 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:35:17.687 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:35:17.687 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:35:17.687 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:35:17.687 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:35:17.688 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:35:17.688 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:17.688 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:35:17.688 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:17.688 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:35:17.689 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:35:17.689 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:35:17.690 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:35:17.690 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:35:17.690 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:35:17.690 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:35:17.691 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:35:17.691 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:35:17.691 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:17.691 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:17.692 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:17.692 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:35:17.692 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:35:17.692 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:35:17.692 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568112.168881_147.1", 2021-09-02T07:35:17.693 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:35:17.693 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:35:17.693 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:35:17.693 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:12.168881Z", 2021-09-02T07:35:17.694 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:35:17.694 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:17.695 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:17.695 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:17.695 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:35:17.695 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:35:17.695 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:35:17.696 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:35:17.696 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:35:17.696 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:35:17.696 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:35:17.697 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:35:17.697 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:35:17.697 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:17.697 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:17.697 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:17.698 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:35:17.698 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:35:17.698 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:35:17.698 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:35:17.699 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:35:17.699 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:35:17.699 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:35:17.700 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:35:17.700 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:35:17.700 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:17.700 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:17.701 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:35:17.701 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:17.702 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:35:17.702 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:35:17.702 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:35:17.702 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:17.703 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:35:17.704 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T07:35:17.705 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T07:35:17.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:17.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:17.707 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:35:17.750 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:17.751 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:17.828 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:35:17.829 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:35:17.829 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:35:17.829 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:35:17.830 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:35:17.830 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:35:17.830 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:17.830 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:17.831 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:35:17.831 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:17.831 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:35:17.832 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:17.832 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:17.833 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:35:17.833 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:17.833 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:35:17.834 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:17.834 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:35:17.834 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:17.835 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:17.835 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:17.835 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:17.836 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:35:17.836 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:17.836 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:17.837 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568112.168881_147.1", 2021-09-02T07:35:17.837 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:17.838 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:17.838 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:17.838 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:12.168881Z", 2021-09-02T07:35:17.839 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:17.839 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:17.839 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:17.840 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:17.840 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:35:17.840 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:17.841 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:17.841 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:17.841 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:17.842 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:17.842 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:17.843 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:17.843 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:17.843 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:17.844 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:17.844 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:17.844 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:35:17.844 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:17.844 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:17.845 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:17.845 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:17.845 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:17.845 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:17.846 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:17.846 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:17.846 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:17.846 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:17.847 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:35:17.847 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:17.847 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:35:17.847 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:35:17.847 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:35:17.848 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:17.848 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:35:17.849 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-02T07:35:18.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:18.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:18.037 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:18.089 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:18.090 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:18.112 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:18.113+0000 7f53e9eb4380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:18.112 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:18.113+0000 7f53e9eb4380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:18.112 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:18.113+0000 7f53e9eb4380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:35:18.113 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:18.113+0000 7f53e9eb4380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:18.147 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:18.148+0000 7f53e9eb4380 1 bucket sync caught up with source: 2021-09-02T07:35:18.147 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000003.8.6, , ] 2021-09-02T07:35:18.147 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000003.8.6, , ] 2021-09-02T07:35:18.148 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:18.148+0000 7f53e9eb4380 0 bucket checkpoint complete 2021-09-02T07:35:18.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:18.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:18.175 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:18.226 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:18.226 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:18.252 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:18.254+0000 7fd2331b5380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:18.253 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:18.254+0000 7fd2331b5380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:18.253 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:18.254+0000 7fd2331b5380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:35:18.253 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:18.254+0000 7fd2331b5380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:18.307 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:18.310+0000 7fd2331b5380 1 bucket sync caught up with source: 2021-09-02T07:35:18.307 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.8.6, , ] 2021-09-02T07:35:18.308 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.8.6, , ] 2021-09-02T07:35:18.308 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:18.310+0000 7fd2331b5380 0 bucket checkpoint complete 2021-09-02T07:35:18.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:18.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:18.341 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:18.392 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:18.392 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:18.416 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:18.417+0000 7f31c157c380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:18.416 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:18.417+0000 7f31c157c380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:18.417 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:18.417+0000 7f31c157c380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:35:18.417 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:18.417+0000 7f31c157c380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:18.456 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:18.457+0000 7f31c157c380 1 bucket sync caught up with source: 2021-09-02T07:35:18.456 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000003.8.6, , ] 2021-09-02T07:35:18.457 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000003.8.6, , ] 2021-09-02T07:35:18.458 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:18.457+0000 7f31c157c380 0 bucket checkpoint complete 2021-09-02T07:35:18.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:18.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:18.480 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:18.529 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:18.530 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:18.552 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:18.555+0000 7f618a070380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:18.553 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:18.555+0000 7f618a070380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:18.553 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:18.555+0000 7f618a070380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:35:18.554 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:18.555+0000 7f618a070380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:18.585 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:18.588+0000 7f618a070380 1 waiting for incremental sync to catch up: 2021-09-02T07:35:18.586 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , , , ] 2021-09-02T07:35:18.586 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.72.12, , ] 2021-09-02T07:35:23.587 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:23.588+0000 7f618a070380 1 bucket sync caught up with source: 2021-09-02T07:35:23.588 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.72.12, , ] 2021-09-02T07:35:23.588 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.72.12, , ] 2021-09-02T07:35:23.588 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:23.588+0000 7f618a070380 0 bucket checkpoint complete 2021-09-02T07:35:23.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:23.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:23.607 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:23.661 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:23.662 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:23.687 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:23.688+0000 7f1deb55c380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:23.687 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:23.688+0000 7f1deb55c380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:23.688 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:23.688+0000 7f1deb55c380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:35:23.688 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:23.688+0000 7f1deb55c380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:23.731 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:23.733+0000 7f1deb55c380 1 bucket sync caught up with source: 2021-09-02T07:35:23.731 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.72.12, , ] 2021-09-02T07:35:23.732 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.72.12, , ] 2021-09-02T07:35:23.732 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:23.733+0000 7f1deb55c380 0 bucket checkpoint complete 2021-09-02T07:35:23.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:23.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:23.751 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:23.800 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:23.801 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:23.824 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:23.826+0000 7fd1723c8380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:23.824 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:23.826+0000 7fd1723c8380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:23.825 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:23.826+0000 7fd1723c8380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:35:23.825 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:23.826+0000 7fd1723c8380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:23.855 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:23.856+0000 7fd1723c8380 1 bucket sync caught up with source: 2021-09-02T07:35:23.855 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.92.12, , ] 2021-09-02T07:35:23.855 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.92.12, , ] 2021-09-02T07:35:23.856 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:23.856+0000 7fd1723c8380 0 bucket checkpoint complete 2021-09-02T07:35:23.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:23.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:23.872 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:23.927 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:23.928 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:23.954 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:23.956+0000 7f61680e4380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:23.954 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:23.956+0000 7f61680e4380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:23.954 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:23.956+0000 7f61680e4380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:35:23.955 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:23.956+0000 7f61680e4380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:23.987 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:23.989+0000 7f61680e4380 1 bucket sync caught up with source: 2021-09-02T07:35:23.987 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000001.92.12, , ] 2021-09-02T07:35:23.988 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000001.92.12, , ] 2021-09-02T07:35:23.988 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:23.989+0000 7f61680e4380 0 bucket checkpoint complete 2021-09-02T07:35:24.003 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-24 zones={test-zone2, test-zone1} 2021-09-02T07:35:24.009 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:24.059 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:24.060 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:24.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:24.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:24.114 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-24 zones={test-zone2, test-zone1} 2021-09-02T07:35:24.114 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-24 zones={test-zone3, test-zone1} 2021-09-02T07:35:24.120 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:24.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:24.168 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:24.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:24.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:24.225 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-24 zones={test-zone3, test-zone1} 2021-09-02T07:35:24.226 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-24 zones={test-zone3, test-zone2} 2021-09-02T07:35:24.231 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:24.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:24.280 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:24.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:24.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:24.338 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-24 zones={test-zone3, test-zone2} 2021-09-02T07:35:24.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:24.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:24.340 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:24.389 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:24.390 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:24.412 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:24.414+0000 7fd67d37b380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:24.412 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:24.414+0000 7fd67d37b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:24.412 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:24.414+0000 7fd67d37b380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:35:24.413 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:24.414+0000 7fd67d37b380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:24.458 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:24.460+0000 7fd67d37b380 1 bucket sync caught up with source: 2021-09-02T07:35:24.458 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000001.23.12, , ] 2021-09-02T07:35:24.459 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000001.23.12, , ] 2021-09-02T07:35:24.459 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:24.460+0000 7fd67d37b380 0 bucket checkpoint complete 2021-09-02T07:35:24.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:24.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:24.480 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:24.529 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:24.529 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:24.552 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:24.554+0000 7fd1b2f42380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:24.552 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:24.554+0000 7fd1b2f42380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:24.553 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:24.554+0000 7fd1b2f42380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:35:24.553 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:24.554+0000 7fd1b2f42380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:24.583 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:24.585+0000 7fd1b2f42380 1 bucket sync caught up with source: 2021-09-02T07:35:24.584 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.23.12, , ] 2021-09-02T07:35:24.584 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.23.12, , ] 2021-09-02T07:35:24.584 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:24.585+0000 7fd1b2f42380 0 bucket checkpoint complete 2021-09-02T07:35:24.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:24.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:24.600 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:24.649 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:24.650 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:24.672 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:24.674+0000 7f5503fdb380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:24.672 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:24.674+0000 7f5503fdb380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:24.672 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:24.674+0000 7f5503fdb380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:35:24.673 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:24.674+0000 7f5503fdb380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:24.716 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:24.719+0000 7f5503fdb380 1 bucket sync caught up with source: 2021-09-02T07:35:24.717 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000001.23.12, , ] 2021-09-02T07:35:24.717 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000001.23.12, , ] 2021-09-02T07:35:24.717 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:24.719+0000 7f5503fdb380 0 bucket checkpoint complete 2021-09-02T07:35:24.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:24.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:24.734 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:24.784 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:24.784 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:24.807 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:24.809+0000 7fe7498bb380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:24.808 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:24.809+0000 7fe7498bb380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:24.808 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:24.809+0000 7fe7498bb380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:35:24.808 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:24.809+0000 7fe7498bb380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:24.852 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:24.853+0000 7fe7498bb380 1 bucket sync caught up with source: 2021-09-02T07:35:24.852 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.326.6, , ] 2021-09-02T07:35:24.852 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.326.6, , ] 2021-09-02T07:35:24.853 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:24.853+0000 7fe7498bb380 0 bucket checkpoint complete 2021-09-02T07:35:24.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:24.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:24.870 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:24.920 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:24.921 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:24.943 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:24.945+0000 7fbde57a7380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:24.943 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:24.945+0000 7fbde57a7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:24.943 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:24.945+0000 7fbde57a7380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:35:24.944 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:24.945+0000 7fbde57a7380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:24.975 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:24.976+0000 7fbde57a7380 1 bucket sync caught up with source: 2021-09-02T07:35:24.975 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.326.6, , ] 2021-09-02T07:35:24.975 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.326.6, , ] 2021-09-02T07:35:24.976 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:24.976+0000 7fbde57a7380 0 bucket checkpoint complete 2021-09-02T07:35:24.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:24.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:24.991 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:25.039 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:25.039 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:25.061 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:25.063+0000 7f44f174e380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:25.061 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:25.063+0000 7f44f174e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:25.062 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:25.063+0000 7f44f174e380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:35:25.062 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:25.063+0000 7f44f174e380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:25.092 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:25.093+0000 7f44f174e380 1 bucket sync caught up with source: 2021-09-02T07:35:25.092 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.338.12, , ] 2021-09-02T07:35:25.092 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.338.12, , ] 2021-09-02T07:35:25.092 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:25.093+0000 7f44f174e380 0 bucket checkpoint complete 2021-09-02T07:35:25.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:25.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:25.108 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:25.157 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:25.158 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:25.179 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:25.181+0000 7f4fced5c380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:25.180 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:25.181+0000 7f4fced5c380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:25.180 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:25.181+0000 7f4fced5c380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:35:25.180 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:25.181+0000 7f4fced5c380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:25.216 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:25.218+0000 7f4fced5c380 1 bucket sync caught up with source: 2021-09-02T07:35:25.216 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000001.338.12, , ] 2021-09-02T07:35:25.216 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000001.338.12, , ] 2021-09-02T07:35:25.217 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:25.218+0000 7f4fced5c380 0 bucket checkpoint complete 2021-09-02T07:35:25.232 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-25 zones={test-zone2, test-zone1} 2021-09-02T07:35:25.238 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:25.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:25.288 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:25.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:25.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:25.346 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-25 zones={test-zone2, test-zone1} 2021-09-02T07:35:25.347 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-25 zones={test-zone3, test-zone1} 2021-09-02T07:35:25.352 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:25.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:25.400 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:25.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:25.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:25.458 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-25 zones={test-zone3, test-zone1} 2021-09-02T07:35:25.459 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-25 zones={test-zone3, test-zone2} 2021-09-02T07:35:25.464 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:25.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:25.512 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:25.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:25.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:25.566 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-25 zones={test-zone3, test-zone2} 2021-09-02T07:35:25.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:25.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:25.567 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:25.617 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:25.617 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:25.640 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:25.643+0000 7f0351612380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:25.641 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:25.643+0000 7f0351612380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:25.641 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:25.643+0000 7f0351612380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:35:25.641 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:25.643+0000 7f0351612380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:25.680 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:25.682+0000 7f0351612380 1 bucket sync caught up with source: 2021-09-02T07:35:25.680 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000001.15.12, , ] 2021-09-02T07:35:25.680 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000001.15.12, , ] 2021-09-02T07:35:25.681 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:25.682+0000 7f0351612380 0 bucket checkpoint complete 2021-09-02T07:35:25.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:25.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:25.699 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:25.747 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:25.748 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:25.775 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:25.777+0000 7fc462312380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:25.776 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:25.777+0000 7fc462312380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:25.776 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:25.777+0000 7fc462312380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:35:25.776 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:25.777+0000 7fc462312380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:25.818 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:25.820+0000 7fc462312380 1 bucket sync caught up with source: 2021-09-02T07:35:25.818 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.15.12, , ] 2021-09-02T07:35:25.818 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.15.12, , ] 2021-09-02T07:35:25.819 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:25.820+0000 7fc462312380 0 bucket checkpoint complete 2021-09-02T07:35:25.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:25.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:25.837 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:25.885 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:25.886 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:25.908 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:25.910+0000 7f60d3526380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:25.909 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:25.910+0000 7f60d3526380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:25.909 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:25.910+0000 7f60d3526380 1 Zone: test-zone4 (a0dd8689-bd06-4f6e-8710-337db82a68b6) 2021-09-02T07:35:25.909 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:25.910+0000 7f60d3526380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:25.949 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:25.951+0000 7f60d3526380 1 bucket sync caught up with source: 2021-09-02T07:35:25.949 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000001.15.12, , ] 2021-09-02T07:35:25.949 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000001.15.12, , ] 2021-09-02T07:35:25.950 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:25.951+0000 7f60d3526380 0 bucket checkpoint complete 2021-09-02T07:35:25.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:25.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:25.967 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:26.016 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:26.016 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:26.038 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:26.040+0000 7feb285ee380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:26.039 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:26.040+0000 7feb285ee380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:26.039 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:26.040+0000 7feb285ee380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:35:26.039 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:26.040+0000 7feb285ee380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:26.076 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:26.077+0000 7feb285ee380 1 bucket sync caught up with source: 2021-09-02T07:35:26.076 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.270.12, , ] 2021-09-02T07:35:26.076 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.270.12, , ] 2021-09-02T07:35:26.076 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:26.077+0000 7feb285ee380 0 bucket checkpoint complete 2021-09-02T07:35:26.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:26.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:26.093 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:26.141 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:26.142 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:26.164 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:26.165+0000 7fa59869c380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:26.165 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:26.165+0000 7fa59869c380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:26.165 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:26.165+0000 7fa59869c380 1 Zone: test-zone3 (5bdffc94-40ec-4bcf-a75a-819f7216b883) 2021-09-02T07:35:26.165 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:26.165+0000 7fa59869c380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:26.195 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:26.197+0000 7fa59869c380 1 bucket sync caught up with source: 2021-09-02T07:35:26.196 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000001.270.12, , ] 2021-09-02T07:35:26.196 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000001.270.12, , ] 2021-09-02T07:35:26.196 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:26.197+0000 7fa59869c380 0 bucket checkpoint complete 2021-09-02T07:35:26.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:26.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:26.212 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket efybpk-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:26.261 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:26.261 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:26.282 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:26.284+0000 7f21d5d21380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:26.283 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:26.284+0000 7f21d5d21380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:26.283 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:26.284+0000 7f21d5d21380 1 Zone: test-zone1 (8b6c886f-f9c4-4c5c-b256-edd3d1362f1c) 2021-09-02T07:35:26.283 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:26.284+0000 7f21d5d21380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:26.315 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:26.316+0000 7f21d5d21380 1 bucket sync caught up with source: 2021-09-02T07:35:26.315 INFO:teuthology.orchestra.run.smithi175.stderr: local status: [, , , , , , , , 00000000003.401.6, , ] 2021-09-02T07:35:26.315 INFO:teuthology.orchestra.run.smithi175.stderr: remote markers: [, , , , , , , , 00000000003.401.6, , ] 2021-09-02T07:35:26.316 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T07:35:26.316+0000 7f21d5d21380 0 bucket checkpoint complete 2021-09-02T07:35:26.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:26.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'efybpk-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-02T07:35:26.334 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket efybpk-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-02T07:35:26.385 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:26.385 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:26.407 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:26.409+0000 7fe7735bf380 1 Realm: test-realm (7cd628c3-b272-4627-b6df-82e8f9eb7cb4) 2021-09-02T07:35:26.408 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:26.409+0000 7fe7735bf380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-02T07:35:26.408 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:26.409+0000 7fe7735bf380 1 Zone: test-zone2 (e43b32ce-d30a-498d-8211-34e9897b4509) 2021-09-02T07:35:26.408 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:26.409+0000 7fe7735bf380 1 using period configuration: 3c6be1ae-90c2-49d2-9d54-568c49b66f2d:4 2021-09-02T07:35:26.451 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:26.453+0000 7fe7735bf380 1 bucket sync caught up with source: 2021-09-02T07:35:26.451 INFO:teuthology.orchestra.run.smithi197.stderr: local status: [, , , , , , , , 00000000003.401.6, , ] 2021-09-02T07:35:26.451 INFO:teuthology.orchestra.run.smithi197.stderr: remote markers: [, , , , , , , , 00000000003.401.6, , ] 2021-09-02T07:35:26.452 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:35:26.453+0000 7fe7735bf380 0 bucket checkpoint complete 2021-09-02T07:35:26.471 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-26 zones={test-zone2, test-zone1} 2021-09-02T07:35:26.477 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:26.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:26.528 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:26.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:26.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:26.586 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-26 zones={test-zone2, test-zone1} 2021-09-02T07:35:26.588 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-26 zones={test-zone3, test-zone1} 2021-09-02T07:35:26.594 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:26.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:26.640 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:26.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:26.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:26.698 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-26 zones={test-zone3, test-zone1} 2021-09-02T07:35:26.699 INFO:tasks.rgw_multi.zone_rados:comparing bucket=efybpk-26 zones={test-zone3, test-zone2} 2021-09-02T07:35:26.704 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-02T07:35:26.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:26.756 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-02T07:35:26.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-02T07:35:26.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-02T07:35:26.814 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=efybpk-26 zones={test-zone3, test-zone2} 2021-09-02T07:35:26.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2021-09-02T07:35:26.815 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-27 2021-09-02T07:35:26.844 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=efybpk-28 2021-09-02T07:35:26.890 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=efybpk-29 2021-09-02T07:35:27.087 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2021-09-02T07:35:27.087 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-30 2021-09-02T07:35:27.106 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=efybpk-31 2021-09-02T07:35:27.146 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=efybpk-32 2021-09-02T07:35:27.359 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2021-09-02T07:35:27.360 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-33 2021-09-02T07:35:27.412 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=efybpk-34 2021-09-02T07:35:27.583 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=efybpk-35 2021-09-02T07:35:29.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2021-09-02T07:35:29.928 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-36 2021-09-02T07:35:29.957 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=efybpk-37 2021-09-02T07:35:30.003 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=efybpk-38 2021-09-02T07:35:30.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:30.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:30.061 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:35:30.118 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:30.118 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:30.247 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T07:35:30.271 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:35:30.271 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:30.272 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:35:30.272 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:32:10.734248Z" 2021-09-02T07:35:30.272 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:30.272 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:30.272 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568130.037580_220.1", 2021-09-02T07:35:30.273 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:35:30.037580Z" 2021-09-02T07:35:30.273 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:30.273 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:30.273 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:35:30.274 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:35:30.274 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:30.274 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:30.274 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:35:30.275 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:35:30.275 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:30.275 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:35:30.277 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T07:35:30.277 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:30.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:30.279 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:35:30.322 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:30.322 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:30.430 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:35:30.430 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:35:30.430 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:35:30.431 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:35:30.431 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:35:30.431 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:35:30.432 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:30.432 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:30.432 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:35:30.432 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:30.433 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:35:30.433 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:30.433 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:30.433 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:35:30.434 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:30.434 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:35:30.434 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:30.434 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:35:30.434 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:30.435 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:30.435 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:30.435 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:30.435 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:35:30.436 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:30.436 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:30.436 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568129.876241_207.1", 2021-09-02T07:35:30.436 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:30.437 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:30.437 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:30.437 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:29.876241Z", 2021-09-02T07:35:30.437 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:30.437 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:30.438 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:30.438 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:30.438 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:35:30.438 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:30.439 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:30.439 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:30.439 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:30.439 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:30.439 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:30.440 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:30.440 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:30.440 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:30.440 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:30.440 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:30.441 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:35:30.441 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:30.441 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:30.441 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:30.442 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:30.443 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:30.443 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:30.443 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:30.444 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:30.444 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:30.444 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:30.445 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:35:30.445 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:30.445 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:35:30.445 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:35:30.446 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:35:30.446 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:30.446 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:35:30.447 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568129.876241_207.1 2021-09-02T07:35:35.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:35.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:35.450 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:35:35.496 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:35.497 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:35.650 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:35:35.651 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:35:35.651 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:35:35.651 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:35:35.651 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:35:35.652 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:35:35.652 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:35.652 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:35.653 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:35:35.653 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:35.653 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:35:35.653 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:35.654 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:35.654 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:35:35.654 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:35.654 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:35:35.655 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:35.655 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:35:35.655 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:35.655 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:35.656 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:35.656 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:35.656 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:35:35.657 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:35.657 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:35.657 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568129.951314_211.1", 2021-09-02T07:35:35.657 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:35.658 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:35.658 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:35.658 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:29.951314Z", 2021-09-02T07:35:35.658 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:35.659 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:35.659 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:35.659 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:35.659 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:35:35.660 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:35.660 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:35.660 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:35.660 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:35.661 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:35.661 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:35.661 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:35.661 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:35.661 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:35.662 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:35.662 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:35.662 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:35:35.663 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:35.663 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:35.663 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:35.663 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:35.664 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:35.664 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:35.664 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:35.664 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:35.665 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:35.665 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:35.665 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:35:35.665 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:35.665 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:35:35.666 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:35:35.666 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:35:35.666 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:35.666 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:35:35.668 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568129.951314_211.1 2021-09-02T07:35:40.669 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:40.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:40.671 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:35:40.721 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:40.721 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:40.872 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:35:40.872 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:35:40.873 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:35:40.873 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:35:40.873 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:35:40.873 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:35:40.874 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:40.874 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:40.874 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:35:40.874 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:40.875 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:35:40.875 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:40.875 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:40.875 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:35:40.876 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:40.876 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:35:40.876 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:40.876 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:35:40.876 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:40.877 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:40.877 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:40.877 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:40.877 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:35:40.878 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:40.878 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:40.878 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568129.876241_207.1", 2021-09-02T07:35:40.878 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:40.879 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:40.879 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:40.879 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:29.876241Z", 2021-09-02T07:35:40.879 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:40.879 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:40.880 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:40.880 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:40.880 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:35:40.880 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:40.880 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:40.881 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:40.881 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:40.881 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:40.881 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:40.882 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:40.882 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:40.882 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:40.882 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:40.883 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:40.883 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:35:40.883 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:40.883 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:40.883 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:40.884 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:40.884 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:40.884 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:40.884 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:40.884 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:40.885 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:40.885 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:40.885 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:35:40.885 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:40.886 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:35:40.886 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:35:40.886 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:35:40.886 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:40.886 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:35:40.887 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568129.876241_207.1 2021-09-02T07:35:45.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:45.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:45.889 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:35:45.931 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:45.932 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:46.001 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:35:46.001 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:35:46.001 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:35:46.002 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:35:46.002 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:35:46.002 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:35:46.002 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:46.003 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:46.003 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:35:46.003 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:46.003 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:35:46.004 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:46.004 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:46.004 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:35:46.004 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:46.004 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:35:46.005 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:46.005 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:35:46.005 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:46.005 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:46.006 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:46.006 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:46.006 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:35:46.007 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:46.007 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:46.007 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568129.944071_209.1", 2021-09-02T07:35:46.007 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:46.007 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:46.008 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:46.008 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:29.944071Z", 2021-09-02T07:35:46.010 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:46.010 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:46.010 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:46.011 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:46.011 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:35:46.011 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:46.012 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:46.012 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:46.012 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:46.012 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:46.013 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:46.013 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:46.013 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:46.013 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:46.013 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:46.014 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:46.014 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:35:46.014 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:46.014 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:46.015 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:46.015 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:46.015 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:46.015 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:46.016 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:46.016 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:46.016 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:46.016 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:46.017 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:35:46.017 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:46.017 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:35:46.017 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:35:46.018 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:35:46.018 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:46.018 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:35:46.020 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568129.944071_209.1 2021-09-02T07:35:51.020 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:51.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:51.022 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:35:51.070 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:51.071 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:51.162 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:35:51.162 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:35:51.163 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:35:51.163 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:35:51.163 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:35:51.163 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:35:51.164 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:51.164 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:51.164 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:35:51.164 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:51.165 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:35:51.165 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:51.165 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:51.165 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:35:51.165 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:51.166 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:35:51.166 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:51.166 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:35:51.166 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:51.167 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:51.167 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:51.167 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:51.167 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:35:51.168 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:51.168 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:51.168 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568130.037580_220.1", 2021-09-02T07:35:51.168 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:51.168 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:51.169 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:51.169 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:30.037580Z", 2021-09-02T07:35:51.169 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:51.169 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:51.170 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:51.170 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:51.170 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:35:51.170 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:51.171 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:51.171 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:51.171 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:51.171 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:51.171 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:51.172 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:51.172 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:51.172 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:51.172 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:51.173 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:35:51.173 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:35:51.173 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:35:51.173 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:35:51.173 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:35:51.174 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:35:51.174 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:35:51.174 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:35:51.174 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:35:51.175 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:35:51.175 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:51.175 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:51.175 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:35:51.176 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:35:51.176 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:35:51.176 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:35:51.176 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:35:51.177 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:35:51.177 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:35:51.178 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T07:35:51.178 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T07:35:51.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:51.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:51.180 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:35:51.226 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:51.226 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:51.304 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:35:51.304 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:35:51.305 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:35:51.305 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:35:51.305 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:35:51.305 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:35:51.306 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:35:51.306 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:51.306 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:35:51.306 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:51.307 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:35:51.307 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:35:51.307 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:35:51.307 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:35:51.308 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:35:51.308 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:35:51.308 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:35:51.308 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:35:51.308 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:35:51.309 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:51.309 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:51.309 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:51.309 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:35:51.310 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:35:51.310 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:35:51.310 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568127.043408_172.1", 2021-09-02T07:35:51.311 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:35:51.311 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:35:51.311 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:35:51.311 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:27.043408Z", 2021-09-02T07:35:51.312 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:35:51.312 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:51.312 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:51.312 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:51.313 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:35:51.313 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:35:51.313 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:35:51.313 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:35:51.313 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:35:51.314 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:35:51.314 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:35:51.314 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:35:51.314 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:35:51.315 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:51.315 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:51.315 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:51.315 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:35:51.316 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:35:51.316 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:35:51.316 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:35:51.316 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:35:51.317 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:35:51.317 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:35:51.317 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:35:51.317 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:35:51.318 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:51.318 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:51.318 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:35:51.318 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:51.318 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:35:51.319 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:35:51.319 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:35:51.319 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:51.319 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:35:51.321 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568127.043408_172.1 2021-09-02T07:35:56.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:56.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:35:56.323 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:35:56.379 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:35:56.379 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:35:56.700 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:35:56.700 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:35:56.701 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:35:56.701 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:35:56.701 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:35:56.701 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:35:56.702 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:35:56.702 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:56.702 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:35:56.702 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:56.703 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:35:56.703 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:35:56.703 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:35:56.703 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:35:56.703 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:35:56.704 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:35:56.704 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:35:56.704 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:35:56.704 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:35:56.705 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:56.705 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:56.705 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:56.705 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:35:56.705 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:35:56.706 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:35:56.706 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568127.043408_172.1", 2021-09-02T07:35:56.706 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:35:56.706 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:35:56.707 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:35:56.707 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:27.043408Z", 2021-09-02T07:35:56.707 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:35:56.707 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:56.707 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:56.708 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:56.708 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:35:56.708 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:35:56.708 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:35:56.709 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:35:56.709 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:35:56.709 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:35:56.709 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:35:56.710 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:35:56.710 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:35:56.710 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:56.710 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:56.710 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:35:56.711 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:35:56.711 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:35:56.711 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:35:56.711 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:35:56.712 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:35:56.712 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:35:56.712 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:35:56.712 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:35:56.712 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:35:56.713 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:56.713 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:56.713 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:35:56.714 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:35:56.714 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:35:56.714 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:35:56.714 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:35:56.715 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:35:56.715 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:35:56.716 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568127.043408_172.1 2021-09-02T07:36:00.009 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:36:00.010+0000 7f968f999700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-02T07:36:00.011 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:36:00.012+0000 7f968f999700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-02T07:36:00.011 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:36:00.012+0000 7f968f999700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-02T07:36:01.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:01.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:01.719 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:36:01.763 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:36:01.763 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:36:01.835 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:36:01.836 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:36:01.836 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:36:01.836 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:36:01.837 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:36:01.837 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:36:01.837 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:01.837 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:01.838 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:36:01.838 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:01.838 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:36:01.838 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:01.838 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:01.839 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:36:01.839 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:01.839 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:36:01.839 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:01.840 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:36:01.840 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:01.840 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:01.840 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:01.840 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:01.841 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:36:01.841 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:01.841 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:01.841 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T07:36:01.842 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:01.842 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:01.842 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:01.842 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T07:36:01.842 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:01.843 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:01.843 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:01.843 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:01.844 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:36:01.844 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:01.844 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:01.844 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:01.845 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:01.845 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:01.845 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:01.845 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:01.845 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:01.846 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:01.846 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:01.847 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:01.847 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:36:01.847 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:01.847 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:01.848 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:01.848 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:01.848 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:01.848 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:01.848 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:01.849 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:01.850 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:01.850 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:01.850 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:36:01.850 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:01.851 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:36:01.851 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:36:01.851 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:36:01.851 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:01.852 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:36:01.853 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.906521_163.1 2021-09-02T07:36:06.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:06.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:06.856 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:36:06.901 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:36:06.901 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:36:06.963 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:36:06.963 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:36:06.964 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:36:06.964 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:36:06.964 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:36:06.964 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:36:06.965 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:06.965 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:06.965 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:36:06.965 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:06.966 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:36:06.966 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:06.966 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:06.966 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:36:06.967 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:06.967 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:36:06.967 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:06.967 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:36:06.968 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:06.968 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:06.969 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:06.969 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:06.969 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:36:06.969 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:06.970 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:06.970 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T07:36:06.970 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:06.970 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:06.971 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:06.971 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T07:36:06.971 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:06.971 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:06.972 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:06.972 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:06.972 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:36:06.972 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:06.973 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:06.973 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:06.973 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:06.973 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:06.974 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:06.974 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:06.974 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:06.974 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:06.975 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:06.975 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:06.975 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:36:06.975 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:06.976 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:06.976 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:06.976 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:06.976 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:06.977 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:06.977 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:06.977 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:06.977 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:06.978 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:06.978 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:36:06.978 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:06.978 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:36:06.978 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:36:06.979 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:36:06.979 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:06.979 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:36:06.980 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.906521_163.1 2021-09-02T07:36:11.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:11.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:11.983 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:36:12.025 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:36:12.025 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:36:12.088 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:36:12.088 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:36:12.088 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:36:12.089 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:36:12.089 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:36:12.089 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:36:12.089 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:12.090 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:12.090 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:36:12.090 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:12.090 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:36:12.091 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:12.091 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:12.091 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:36:12.091 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:12.092 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:36:12.092 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:12.092 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:36:12.092 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:12.093 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:12.093 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:12.093 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:12.093 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:36:12.094 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:12.094 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:12.094 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568127.039849_171.1", 2021-09-02T07:36:12.094 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:12.095 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:12.095 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:12.095 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:27.039849Z", 2021-09-02T07:36:12.095 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:12.096 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:12.096 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:12.096 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:12.096 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:36:12.097 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:12.097 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:12.097 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:12.098 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:12.098 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:12.098 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:12.098 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:12.099 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:12.099 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:12.099 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:12.099 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:12.100 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:36:12.100 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:12.100 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:12.100 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:12.101 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:12.101 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:12.101 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:12.101 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:12.102 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:12.102 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:12.102 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:12.103 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:36:12.103 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:12.103 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:36:12.103 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:36:12.104 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:36:12.104 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:12.104 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:36:12.105 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568127.039849_171.1 2021-09-02T07:36:17.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:17.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:17.108 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:36:17.151 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:36:17.151 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:36:17.214 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:36:17.214 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:36:17.214 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:36:17.215 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:36:17.215 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:36:17.215 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:36:17.215 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:17.216 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:17.216 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:36:17.216 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:17.216 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:36:17.217 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:17.217 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:17.217 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:36:17.217 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:17.217 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:36:17.218 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:17.218 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:36:17.218 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:17.218 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:17.219 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:17.219 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:17.219 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:36:17.219 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:17.220 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:17.220 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T07:36:17.220 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:17.220 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:17.221 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:17.221 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T07:36:17.221 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:17.221 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:17.222 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:17.222 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:17.222 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:36:17.222 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:17.223 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:17.223 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:17.223 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:17.223 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:17.223 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:17.224 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:17.224 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:17.224 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:17.224 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:17.225 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:17.225 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:36:17.225 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:17.225 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:17.225 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:17.226 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:17.226 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:17.226 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:17.226 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:17.227 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:17.227 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:17.227 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:17.228 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:36:17.228 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:17.228 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:36:17.228 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:36:17.229 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:36:17.229 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:17.229 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:36:17.230 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.906521_163.1 2021-09-02T07:36:22.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:22.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:22.234 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:36:22.278 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:36:22.279 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:36:22.499 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:36:22.500 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:36:22.500 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:36:22.500 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:36:22.500 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:36:22.501 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:36:22.501 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:22.501 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:22.502 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:36:22.502 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:22.502 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:36:22.502 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:22.503 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:22.503 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:36:22.503 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:22.503 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:36:22.503 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:22.504 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:36:22.504 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:22.504 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:22.504 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:22.505 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:22.506 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:36:22.506 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:22.506 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:22.507 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568126.911309_165.1", 2021-09-02T07:36:22.507 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:22.508 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:22.508 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:22.508 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:26.911309Z", 2021-09-02T07:36:22.508 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:22.509 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:22.509 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:22.509 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:22.509 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:36:22.510 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:22.510 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:22.510 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:22.510 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:22.510 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:22.511 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:22.511 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:22.511 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:22.511 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:22.512 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:22.512 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:22.512 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:36:22.512 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:22.512 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:22.513 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:22.513 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:22.513 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:22.513 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:22.514 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:22.514 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:22.514 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:22.514 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:22.515 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:36:22.515 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:22.515 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:36:22.515 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:36:22.515 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:36:22.516 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:22.516 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:36:22.517 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.911309_165.1 2021-09-02T07:36:27.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:27.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:27.520 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:36:27.563 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:36:27.564 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:36:27.637 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:36:27.637 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:36:27.637 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:36:27.638 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:36:27.639 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:36:27.639 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:36:27.639 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:27.639 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:27.640 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:36:27.640 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:27.640 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:36:27.641 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:27.641 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:27.641 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:36:27.641 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:27.642 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:36:27.642 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:27.642 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:36:27.642 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:27.642 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:27.643 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:27.643 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:27.644 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:36:27.644 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:27.644 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:27.644 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T07:36:27.644 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:27.645 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:27.645 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:27.645 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T07:36:27.645 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:27.646 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:27.646 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:27.646 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:27.646 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:36:27.647 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:27.647 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:27.647 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:27.647 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:27.647 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:27.648 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:27.648 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:27.648 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:27.648 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:27.649 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:27.649 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:27.649 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:36:27.649 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:27.649 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:27.650 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:27.650 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:27.650 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:27.650 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:27.651 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:27.651 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:27.651 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:27.651 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:27.651 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:36:27.652 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:27.652 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:36:27.652 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:36:27.652 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:36:27.653 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:27.653 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:36:27.654 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.906521_163.1 2021-09-02T07:36:32.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:32.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:32.657 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:36:32.700 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:36:32.701 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:36:32.778 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:36:32.778 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:36:32.779 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:36:32.780 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:36:32.781 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:36:32.781 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:36:32.781 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:32.781 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:32.782 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:36:32.782 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:32.782 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:36:32.782 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:32.783 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:32.783 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:36:32.783 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:32.783 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:36:32.784 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:32.784 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:36:32.784 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:32.784 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:32.784 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:32.785 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:32.785 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:36:32.785 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:32.785 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:32.786 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568126.985199_169.1", 2021-09-02T07:36:32.786 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:32.786 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:32.787 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:32.787 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:26.985199Z", 2021-09-02T07:36:32.787 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:32.787 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:32.787 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:32.788 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:32.788 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:36:32.788 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:32.788 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:32.789 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:32.789 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:32.789 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:32.789 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:32.789 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:32.790 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:32.790 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:32.790 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:32.790 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:32.791 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:36:32.791 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:32.791 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:32.791 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:32.792 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:32.792 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:32.792 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:32.792 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:32.793 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:32.793 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:32.793 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:32.793 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:36:32.794 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:32.794 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:36:32.794 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:36:32.794 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:36:32.795 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:32.795 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:36:32.796 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.985199_169.1 2021-09-02T07:36:37.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:37.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:37.799 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:36:37.841 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:36:37.841 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:36:37.901 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:36:37.901 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:36:37.902 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:36:37.903 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:36:37.903 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:36:37.903 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:36:37.904 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:37.904 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:37.904 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:36:37.904 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:37.905 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:36:37.905 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:37.905 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:37.905 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:36:37.905 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:37.906 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:36:37.906 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:37.906 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:36:37.906 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:37.907 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:37.907 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:37.907 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:37.907 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:36:37.908 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:37.908 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:37.909 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:36:37.909 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:37.910 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:37.910 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:37.910 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:36:37.911 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:37.911 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:37.912 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:37.912 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:37.912 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:36:37.913 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:37.913 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:37.914 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:37.915 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:37.915 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:37.915 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:37.916 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:37.916 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:37.917 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:37.917 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:37.917 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:37.918 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:36:37.918 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:37.918 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:37.920 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:37.920 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:37.921 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:37.921 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:37.922 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:37.922 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:37.922 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:37.923 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:37.923 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:36:37.924 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:37.924 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:36:37.925 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:36:37.925 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:36:37.926 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:37.926 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:36:37.928 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:36:42.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:42.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:42.931 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:36:42.974 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:36:42.975 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:36:43.035 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:36:43.035 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:36:43.035 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:36:43.036 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:36:43.036 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:36:43.036 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:36:43.036 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:43.037 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:43.037 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:36:43.037 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:43.037 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:36:43.038 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:43.038 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:43.038 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:36:43.038 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:43.038 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:36:43.039 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:43.039 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:36:43.039 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:43.039 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:43.040 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:43.040 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:43.040 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:36:43.040 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:43.041 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:43.041 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568126.862512_159.1", 2021-09-02T07:36:43.041 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:43.041 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:43.041 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:43.042 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:26.862512Z", 2021-09-02T07:36:43.042 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:43.042 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:43.042 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:43.043 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:43.043 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:36:43.043 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:43.043 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:43.044 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:43.044 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:43.044 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:43.044 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:43.045 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:43.045 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:43.045 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:43.045 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:43.046 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:43.046 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:36:43.046 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:43.046 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:43.046 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:43.047 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:43.047 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:43.047 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:43.047 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:43.048 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:43.048 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:43.048 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:43.048 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:36:43.049 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:43.049 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:36:43.049 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:36:43.049 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:36:43.049 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:43.050 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:36:43.055 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.862512_159.1 2021-09-02T07:36:48.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:48.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:48.058 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:36:48.101 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:36:48.101 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:36:48.164 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:36:48.164 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:36:48.164 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:36:48.165 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:36:48.165 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:36:48.165 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:36:48.165 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:48.165 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:48.166 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:36:48.166 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:48.166 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:36:48.166 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:48.167 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:48.167 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:36:48.167 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:48.167 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:36:48.168 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:48.168 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:36:48.168 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:48.168 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:48.168 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:48.169 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:48.169 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:36:48.169 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:48.169 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:48.170 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568126.914475_166.1", 2021-09-02T07:36:48.170 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:48.170 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:48.170 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:48.171 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:26.914475Z", 2021-09-02T07:36:48.171 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:48.171 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:48.171 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:48.171 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:48.172 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:36:48.172 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:48.172 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:48.172 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:48.172 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:48.173 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:48.173 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:48.173 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:48.173 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:48.173 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:48.174 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:48.174 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:48.174 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:36:48.174 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:48.175 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:48.175 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:48.175 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:48.175 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:48.175 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:48.176 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:48.176 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:48.176 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:48.176 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:48.176 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:36:48.177 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:48.177 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:36:48.177 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:36:48.177 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:36:48.177 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:48.178 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:36:48.179 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.914475_166.1 2021-09-02T07:36:53.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:53.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:53.182 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:36:53.225 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:36:53.225 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:36:53.297 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:36:53.298 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:36:53.298 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:36:53.298 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:36:53.298 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:36:53.299 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:36:53.299 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:53.299 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:53.299 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:36:53.300 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:53.300 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:36:53.300 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:53.300 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:53.301 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:36:53.301 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:53.301 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:36:53.301 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:53.302 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:36:53.302 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:53.302 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:53.302 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:53.302 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:53.303 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:36:53.303 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:53.303 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:53.303 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T07:36:53.304 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:53.304 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:53.304 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:53.304 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T07:36:53.305 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:53.305 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:53.305 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:53.305 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:53.306 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:36:53.306 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:53.306 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:53.306 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:53.307 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:53.307 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:53.307 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:53.307 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:53.308 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:53.308 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:53.308 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:53.308 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:53.309 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:36:53.309 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:53.309 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:53.309 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:53.310 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:53.310 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:53.310 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:53.310 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:53.311 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:53.311 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:53.311 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:53.311 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:36:53.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:53.312 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:36:53.312 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:36:53.312 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:36:53.312 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:53.313 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:36:53.314 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.906521_163.1 2021-09-02T07:36:58.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:58.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:36:58.317 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:36:58.361 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:36:58.362 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:36:58.430 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:36:58.430 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:36:58.431 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:36:58.431 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:36:58.431 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:36:58.431 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:36:58.432 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:58.432 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:58.432 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:36:58.432 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:58.433 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:36:58.433 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:58.433 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:58.433 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:36:58.433 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:58.434 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:36:58.434 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:58.434 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:36:58.434 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:58.435 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:58.435 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:58.435 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:58.435 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:36:58.435 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:58.436 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:58.436 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568127.039849_171.1", 2021-09-02T07:36:58.436 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:58.436 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:58.437 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:58.437 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:27.039849Z", 2021-09-02T07:36:58.437 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:58.437 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:58.437 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:58.438 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:58.438 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:36:58.438 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:58.438 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:58.439 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:58.439 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:58.439 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:58.439 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:58.439 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:58.440 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:58.440 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:58.440 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:58.440 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:36:58.440 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:36:58.441 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:36:58.441 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:36:58.441 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:36:58.441 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:36:58.442 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:36:58.442 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:36:58.442 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:36:58.442 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:36:58.442 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:58.443 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:58.443 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:36:58.443 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:36:58.443 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:36:58.444 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:36:58.444 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:36:58.444 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:36:58.444 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:36:58.446 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568127.039849_171.1 2021-09-02T07:37:03.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:03.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:03.448 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:37:03.501 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:37:03.501 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:37:03.579 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:37:03.580 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:37:03.580 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:37:03.580 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:37:03.580 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:37:03.581 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:37:03.581 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:03.581 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:03.581 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:37:03.582 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:03.582 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:37:03.582 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:03.582 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:03.583 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:37:03.583 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:03.583 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:37:03.583 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:03.584 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:37:03.584 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:03.584 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:03.584 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:03.584 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:03.585 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:37:03.585 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:03.585 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:03.585 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T07:37:03.586 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:03.586 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:03.586 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:03.586 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T07:37:03.586 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:03.587 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:03.587 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:03.587 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:03.587 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:37:03.588 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:03.588 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:03.588 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:37:03.589 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:03.589 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:03.589 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:03.589 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:37:03.590 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:03.590 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:03.590 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:03.590 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:03.591 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:37:03.591 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:03.591 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:03.591 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:37:03.591 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:03.592 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:03.592 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:03.592 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:37:03.592 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:03.593 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:03.593 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:03.593 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:37:03.593 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:03.594 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:37:03.594 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:37:03.594 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:37:03.594 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:03.594 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:37:03.596 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.906521_163.1 2021-09-02T07:37:08.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:08.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:08.600 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:37:08.643 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:37:08.643 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:37:08.705 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:37:08.705 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:37:08.706 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:37:08.706 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:37:08.706 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:37:08.707 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:37:08.707 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:08.707 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:08.707 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:37:08.708 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:08.708 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:37:08.708 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:08.708 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:08.709 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:37:08.709 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:08.709 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:37:08.709 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:08.710 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:37:08.710 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:08.710 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:08.710 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:08.711 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:08.711 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:37:08.711 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:08.711 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:08.712 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T07:37:08.712 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:08.712 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:08.712 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:08.712 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T07:37:08.713 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:08.713 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:08.713 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:08.713 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:08.713 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:37:08.714 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:08.714 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:08.714 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:37:08.714 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:08.715 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:08.715 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:08.715 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:37:08.715 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:08.716 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:08.716 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:08.716 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:08.716 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:37:08.717 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:08.717 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:08.717 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:37:08.717 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:08.717 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:08.718 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:08.718 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:37:08.718 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:08.718 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:08.719 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:08.719 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:37:08.719 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:08.719 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:37:08.720 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:37:08.720 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:37:08.720 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:08.720 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:37:08.721 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568127.103991_175.1 2021-09-02T07:37:13.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:13.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:13.724 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:37:13.792 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:37:13.792 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:37:13.950 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:37:13.950 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:37:13.951 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:37:13.952 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:37:13.952 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:37:13.952 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:37:13.952 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:13.953 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:13.953 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:37:13.953 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:13.953 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:37:13.953 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:13.954 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:13.954 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:37:13.954 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:13.954 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:37:13.955 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:13.955 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:37:13.955 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:13.955 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:13.955 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:13.956 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:13.956 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:37:13.956 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:13.956 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:13.957 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T07:37:13.957 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:13.957 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:13.957 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:13.958 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T07:37:13.958 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:13.958 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:13.958 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:13.959 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:13.959 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:37:13.959 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:13.959 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:13.959 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:37:13.960 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:13.960 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:13.960 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:13.960 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:37:13.961 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:13.961 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:13.961 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:13.961 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:13.961 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:37:13.962 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:13.962 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:13.962 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:37:13.962 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:13.963 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:13.963 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:13.963 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:37:13.964 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:13.964 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:13.964 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:13.964 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:37:13.965 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:13.965 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:37:13.965 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:37:13.965 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:37:13.965 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:13.966 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:37:13.967 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568127.103991_175.1 2021-09-02T07:37:18.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:18.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:18.970 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:37:19.046 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:37:19.046 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:37:19.156 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:37:19.156 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:37:19.157 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:37:19.157 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:37:19.157 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:37:19.157 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:37:19.158 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:19.158 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:19.158 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:37:19.158 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:19.159 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:37:19.159 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:19.159 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:19.159 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:37:19.160 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:19.160 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:37:19.160 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:19.161 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:37:19.161 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:19.161 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:19.161 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:19.162 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:19.162 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:37:19.162 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:19.163 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:19.163 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568126.869904_161.1", 2021-09-02T07:37:19.163 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:19.163 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:19.164 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:19.164 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:26.869904Z", 2021-09-02T07:37:19.164 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:19.164 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:19.165 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:19.165 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:19.165 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:37:19.166 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:19.166 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:19.166 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:37:19.167 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:19.167 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:19.167 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:19.167 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:37:19.167 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:19.168 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:19.168 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:19.168 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:19.168 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:37:19.169 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:19.169 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:19.169 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:37:19.170 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:19.170 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:19.170 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:19.170 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:37:19.171 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:19.171 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:19.171 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:19.172 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:37:19.172 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:19.172 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:37:19.172 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:37:19.173 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:37:19.173 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:19.173 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:37:19.175 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.869904_161.1 2021-09-02T07:37:24.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:24.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:24.178 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:37:24.220 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:37:24.220 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:37:24.297 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:37:24.297 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:37:24.297 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:37:24.298 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:37:24.298 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:37:24.298 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:37:24.298 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:24.298 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:24.299 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:37:24.299 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:24.299 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:37:24.299 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:24.300 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:24.300 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:37:24.300 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:24.300 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:37:24.301 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:24.301 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:37:24.301 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:24.301 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:24.301 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:24.302 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:24.302 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:37:24.302 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:24.302 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:24.303 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568126.869904_161.1", 2021-09-02T07:37:24.303 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:24.303 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:24.303 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:24.303 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:26.869904Z", 2021-09-02T07:37:24.304 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:24.304 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:24.304 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:24.304 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:24.305 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:37:24.305 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:24.305 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:24.305 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:37:24.305 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:24.306 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:24.306 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:24.306 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:37:24.306 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:24.307 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:24.307 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:24.307 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:24.307 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:37:24.308 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:24.308 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:24.308 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:37:24.308 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:24.308 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:24.309 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:24.309 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:37:24.309 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:24.309 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:24.309 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:24.310 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:37:24.310 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:24.310 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:37:24.310 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:37:24.311 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:37:24.311 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:24.311 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:37:24.312 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.869904_161.1 2021-09-02T07:37:29.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:29.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:29.315 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:37:29.359 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:37:29.359 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:37:29.428 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:37:29.429 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:37:29.429 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:37:29.429 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:37:29.429 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:37:29.430 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:37:29.430 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:29.430 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:29.430 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:37:29.431 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:29.431 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:37:29.431 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:29.431 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:29.431 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:37:29.432 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:29.432 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:37:29.432 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:29.432 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:37:29.433 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:29.433 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:29.433 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:29.433 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:29.434 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:37:29.434 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:29.434 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:29.434 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568126.869904_161.1", 2021-09-02T07:37:29.434 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:29.435 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:29.435 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:29.435 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:26.869904Z", 2021-09-02T07:37:29.435 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:29.436 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:29.436 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:29.436 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:29.436 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:37:29.437 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:29.437 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:29.437 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:37:29.437 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:29.437 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:29.438 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:29.438 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:37:29.438 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:29.438 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:29.439 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:29.439 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:29.439 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:37:29.439 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:29.440 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:29.440 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:37:29.440 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:29.440 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:29.440 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:29.441 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:37:29.441 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:29.441 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:29.441 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:29.442 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:37:29.442 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:29.442 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:37:29.442 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:37:29.443 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:37:29.443 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:29.443 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:37:29.444 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.869904_161.1 2021-09-02T07:37:34.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:34.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:34.448 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:37:34.490 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:37:34.490 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:37:34.563 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:37:34.564 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:37:34.564 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:37:34.564 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:37:34.564 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:37:34.565 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:37:34.565 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:34.565 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:34.565 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:37:34.565 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:34.566 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:37:34.566 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:34.566 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:34.566 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:37:34.567 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:34.567 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:37:34.567 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:34.567 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:37:34.568 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:34.568 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:34.568 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:34.568 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:34.568 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:37:34.569 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:34.569 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:34.569 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568126.932719_168.1", 2021-09-02T07:37:34.570 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:34.570 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:34.570 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:34.571 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:26.932719Z", 2021-09-02T07:37:34.571 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:34.571 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:34.571 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:34.572 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:34.572 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:37:34.572 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:34.572 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:34.573 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:37:34.573 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:34.573 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:34.573 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:34.573 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:37:34.574 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:34.574 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:34.574 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:34.574 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:34.575 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:37:34.575 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:34.575 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:34.575 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:37:34.576 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:34.576 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:34.576 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:34.577 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:37:34.577 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:34.577 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:34.577 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:34.577 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:37:34.578 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:34.578 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:37:34.578 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:37:34.579 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:37:34.579 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:34.579 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:37:34.581 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.932719_168.1 2021-09-02T07:37:39.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:39.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:39.584 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:37:39.629 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:37:39.629 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:37:39.694 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:37:39.695 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:37:39.695 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:37:39.695 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:37:39.696 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:37:39.696 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:37:39.696 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:39.696 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:39.697 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:37:39.697 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:39.697 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:37:39.697 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:39.698 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:39.698 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:37:39.698 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:39.698 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:37:39.699 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:39.699 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:37:39.699 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:39.699 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:39.699 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:39.700 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:39.700 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:37:39.700 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:39.700 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:39.701 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568126.932719_168.1", 2021-09-02T07:37:39.701 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:39.701 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:39.701 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:39.702 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:26.932719Z", 2021-09-02T07:37:39.702 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:39.702 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:39.702 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:39.702 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:39.703 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:37:39.703 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:39.703 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:39.703 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:37:39.704 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:39.704 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:39.705 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:39.705 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:37:39.705 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:39.705 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:39.705 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:39.706 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:39.706 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:37:39.706 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:39.706 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:39.707 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:37:39.707 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:39.707 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:39.707 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:39.708 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:37:39.708 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:39.708 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:39.708 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:39.708 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:37:39.709 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:39.709 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:37:39.709 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:37:39.709 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:37:39.710 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:39.710 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:37:39.711 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.932719_168.1 2021-09-02T07:37:40.947 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2021-09-02T07:37:40.947+0000 7f04206f5700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-02T07:37:44.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:44.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:44.714 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:37:44.761 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:37:44.762 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:37:44.830 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:37:44.831 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:37:44.831 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:37:44.831 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:37:44.831 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:37:44.832 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:37:44.832 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:44.832 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:44.832 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:37:44.833 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:44.833 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:37:44.833 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:44.833 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:44.834 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:37:44.834 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:44.834 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:37:44.834 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:44.834 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:37:44.835 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:44.835 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:44.835 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:44.835 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:44.836 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:37:44.836 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:44.836 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:44.836 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568130.037580_220.1", 2021-09-02T07:37:44.836 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:44.837 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:44.837 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:44.837 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:35:30.037580Z", 2021-09-02T07:37:44.837 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:44.838 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:44.838 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:44.838 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:44.838 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:37:44.839 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:44.839 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:44.839 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:37:44.839 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:44.840 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:44.840 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:44.840 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:37:44.840 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:44.840 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:44.841 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:44.841 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:37:44.841 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:37:44.841 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:37:44.842 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:37:44.842 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:37:44.842 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:37:44.842 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:37:44.842 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:37:44.843 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:37:44.843 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:37:44.843 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:44.844 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:44.844 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:37:44.844 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:37:44.844 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:37:44.845 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:37:44.845 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:37:44.845 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:37:44.845 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:37:44.847 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T07:37:44.847 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T07:37:44.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:44.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:44.849 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:37:44.933 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:37:44.934 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:37:45.043 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:37:45.044 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:37:45.044 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:37:45.045 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:37:45.045 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:37:45.045 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:37:45.045 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:37:45.045 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:37:45.046 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:37:45.046 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:37:45.046 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:37:45.046 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:37:45.047 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:37:45.047 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:37:45.047 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:37:45.047 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:37:45.047 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:37:45.048 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:37:45.048 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:37:45.048 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:37:45.048 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:37:45.049 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:37:45.049 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:37:45.049 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:37:45.049 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:37:45.049 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.835437_156.1", 2021-09-02T07:37:45.050 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:37:45.050 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:37:45.050 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:37:45.050 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.835437Z", 2021-09-02T07:37:45.051 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:37:45.051 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:37:45.051 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:37:45.051 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:37:45.052 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:37:45.052 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:37:45.052 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:37:45.052 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:37:45.053 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:37:45.053 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:37:45.053 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:37:45.053 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:37:45.054 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:37:45.054 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:37:45.054 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:37:45.054 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:37:45.054 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:37:45.055 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:37:45.055 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:37:45.055 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:37:45.056 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:37:45.056 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:37:45.056 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:37:45.056 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:37:45.056 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:37:45.057 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:37:45.057 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:37:45.057 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:37:45.057 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:37:45.057 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:37:45.058 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:37:45.058 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:37:45.058 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:37:45.058 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:37:45.060 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.835437_156.1 2021-09-02T07:37:50.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:50.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:50.064 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:37:50.113 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:37:50.114 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:37:50.175 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:37:50.176 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:37:50.176 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:37:50.176 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:37:50.176 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:37:50.177 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:37:50.177 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:37:50.177 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:37:50.177 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:37:50.178 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:37:50.178 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:37:50.178 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:37:50.178 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:37:50.178 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:37:50.179 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:37:50.179 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:37:50.179 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:37:50.179 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:37:50.180 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:37:50.180 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:37:50.180 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:37:50.180 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:37:50.180 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:37:50.181 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:37:50.181 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:37:50.181 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:37:50.181 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:37:50.182 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:37:50.182 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:37:50.182 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:37:50.182 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:37:50.182 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:37:50.183 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:37:50.183 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:37:50.183 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:37:50.183 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:37:50.184 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:37:50.184 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:37:50.184 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:37:50.184 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:37:50.184 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:37:50.185 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:37:50.185 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:37:50.185 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:37:50.185 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:37:50.186 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:37:50.186 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:37:50.186 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:37:50.186 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:37:50.187 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:37:50.187 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:37:50.187 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:37:50.187 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:37:50.187 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:37:50.188 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:37:50.188 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:37:50.188 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:37:50.188 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:37:50.189 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:37:50.189 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:37:50.189 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:37:50.189 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:37:50.189 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:37:50.190 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:37:50.191 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:37:55.192 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:55.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:37:55.193 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:37:55.254 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:37:55.255 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:37:55.321 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:37:55.322 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:37:55.322 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:37:55.322 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:37:55.322 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:37:55.323 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:37:55.323 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:37:55.323 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:37:55.323 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:37:55.324 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:37:55.324 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:37:55.324 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:37:55.324 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:37:55.325 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:37:55.325 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:37:55.325 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:37:55.325 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:37:55.326 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:37:55.326 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:37:55.326 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:37:55.326 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:37:55.326 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:37:55.327 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:37:55.327 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:37:55.327 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:37:55.327 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:37:55.328 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:37:55.328 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:37:55.328 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:37:55.328 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:37:55.328 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:37:55.329 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:37:55.329 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:37:55.329 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:37:55.329 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:37:55.330 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:37:55.330 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:37:55.330 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:37:55.330 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:37:55.331 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:37:55.331 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:37:55.331 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:37:55.331 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:37:55.332 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:37:55.332 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:37:55.332 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:37:55.332 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:37:55.333 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:37:55.333 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:37:55.333 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:37:55.333 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:37:55.334 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:37:55.334 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:37:55.334 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:37:55.334 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:37:55.335 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:37:55.335 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:37:55.335 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:37:55.335 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:37:55.335 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:37:55.336 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:37:55.336 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:37:55.336 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:37:55.336 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:37:55.338 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:38:00.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:00.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:00.340 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:38:00.383 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:38:00.383 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:38:00.458 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:38:00.458 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:38:00.459 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:38:00.459 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:38:00.459 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:38:00.459 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:38:00.459 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:00.460 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:00.460 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:38:00.460 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:00.460 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:38:00.461 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:00.461 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:00.461 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:38:00.461 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:00.462 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:38:00.462 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:00.462 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:38:00.462 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:00.463 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:00.463 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:00.463 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:00.463 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:38:00.464 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:00.464 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:00.464 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.844214_158.1", 2021-09-02T07:38:00.464 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:00.465 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:00.465 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:00.465 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.844214Z", 2021-09-02T07:38:00.465 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:00.465 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:00.466 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:00.466 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:00.466 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:38:00.467 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:00.467 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:00.467 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:00.468 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:00.468 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:00.468 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:00.468 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:00.469 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:00.469 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:00.469 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:00.469 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:00.470 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:38:00.470 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:00.470 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:00.470 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:00.470 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:00.471 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:00.471 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:00.471 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:00.471 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:00.472 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:00.472 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:00.472 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:38:00.472 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:00.473 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:38:00.473 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:38:00.473 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:38:00.473 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:00.473 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:38:00.475 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.844214_158.1 2021-09-02T07:38:05.476 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:05.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:05.478 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:38:05.521 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:38:05.521 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:38:05.596 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:38:05.596 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:38:05.597 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:38:05.597 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:38:05.597 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:38:05.597 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:38:05.597 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:05.598 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:05.598 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:38:05.598 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:05.599 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:38:05.599 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:05.599 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:05.599 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:38:05.600 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:05.600 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:38:05.600 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:05.600 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:38:05.600 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:05.601 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:05.601 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:05.601 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:05.601 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:38:05.602 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:05.602 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:05.602 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.862512_159.1", 2021-09-02T07:38:05.602 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:05.602 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:05.603 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:05.603 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.862512Z", 2021-09-02T07:38:05.603 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:05.603 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:05.604 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:05.604 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:05.604 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:38:05.604 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:05.604 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:05.605 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:05.605 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:05.605 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:05.605 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:05.606 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:05.606 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:05.606 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:05.606 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:05.606 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:05.607 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:38:05.607 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:05.607 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:05.607 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:05.608 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:05.608 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:05.608 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:05.608 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:05.608 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:05.609 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:05.609 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:05.609 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:38:05.609 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:05.610 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:38:05.610 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:38:05.610 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:38:05.610 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:05.611 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:38:05.612 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.862512_159.1 2021-09-02T07:38:10.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:10.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:10.614 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:38:10.659 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:38:10.660 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:38:10.738 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:38:10.738 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:38:10.739 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:38:10.739 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:38:10.739 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:38:10.741 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:38:10.741 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:10.742 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:10.742 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:38:10.742 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:10.742 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:38:10.743 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:10.743 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:10.743 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:38:10.743 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:10.744 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:38:10.744 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:10.744 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:38:10.744 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:10.745 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:10.745 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:10.745 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:10.745 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:38:10.745 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:10.746 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:10.746 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.835437_156.1", 2021-09-02T07:38:10.746 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:10.746 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:10.747 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:10.747 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.835437Z", 2021-09-02T07:38:10.747 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:10.747 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:10.747 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:10.748 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:10.748 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:38:10.748 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:10.748 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:10.749 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:10.749 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:10.749 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:10.749 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:10.749 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:10.750 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:10.750 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:10.750 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:10.750 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:10.751 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:38:10.751 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:10.751 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:10.751 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:10.752 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:10.752 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:10.752 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:10.752 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:10.753 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:10.753 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:10.753 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:10.753 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:38:10.753 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:10.754 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:38:10.754 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:38:10.754 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:38:10.754 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:10.755 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:38:10.756 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.835437_156.1 2021-09-02T07:38:13.934 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:38:13.936+0000 7f968e196700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-02T07:38:15.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:15.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:15.758 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:38:15.806 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:38:15.807 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:38:15.896 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:38:15.897 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:38:15.897 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:38:15.897 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:38:15.898 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:38:15.898 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:38:15.898 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:15.898 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:15.899 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:38:15.899 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:15.899 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:38:15.899 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:15.900 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:15.900 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:38:15.900 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:15.900 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:38:15.901 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:15.901 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:38:15.901 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:15.901 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:15.901 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:15.902 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:15.902 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:38:15.902 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:15.902 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:15.903 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T07:38:15.903 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:15.903 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:15.903 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:15.903 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T07:38:15.904 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:15.904 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:15.904 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:15.904 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:15.905 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:38:15.905 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:15.905 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:15.905 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:15.905 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:15.906 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:15.906 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:15.906 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:15.906 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:15.907 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:15.907 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:15.907 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:15.907 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:38:15.907 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:15.908 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:15.908 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:15.908 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:15.908 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:15.909 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:15.909 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:15.909 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:15.909 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:15.909 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:15.910 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:38:15.910 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:15.910 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:38:15.910 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:38:15.911 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:38:15.911 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:15.911 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:38:15.912 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.906521_163.1 2021-09-02T07:38:20.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:20.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:20.915 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:38:20.963 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:38:20.963 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:38:21.033 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:38:21.033 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:38:21.033 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:38:21.034 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:38:21.034 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:38:21.034 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:38:21.035 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:21.035 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:21.035 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:38:21.035 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:21.036 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:38:21.036 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:21.036 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:21.036 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:38:21.036 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:21.037 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:38:21.037 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:21.037 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:38:21.037 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:21.038 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:21.038 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:21.038 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:21.038 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:38:21.039 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:21.039 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:21.039 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.869904_161.1", 2021-09-02T07:38:21.039 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:21.040 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:21.040 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:21.040 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.869904Z", 2021-09-02T07:38:21.040 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:21.040 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:21.041 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:21.041 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:21.041 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:38:21.041 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:21.042 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:21.042 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:21.042 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:21.042 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:21.043 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:21.043 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:21.043 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:21.043 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:21.043 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:21.044 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:21.044 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:38:21.044 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:21.044 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:21.045 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:21.045 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:21.045 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:21.045 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:21.046 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:21.046 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:21.046 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:21.047 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:21.047 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:38:21.047 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:21.047 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:38:21.047 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:38:21.048 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:38:21.048 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:21.048 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:38:21.049 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.869904_161.1 2021-09-02T07:38:26.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:26.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:26.053 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:38:26.095 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:38:26.096 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:38:26.170 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:38:26.170 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:38:26.171 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:38:26.171 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:38:26.171 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:38:26.171 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:38:26.172 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:26.172 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:26.172 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:38:26.172 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:26.173 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:38:26.173 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:26.173 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:26.173 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:38:26.174 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:26.174 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:38:26.174 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:26.174 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:38:26.175 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:26.175 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:26.175 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:26.175 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:26.176 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:38:26.176 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:26.176 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:26.177 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:38:26.177 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:26.177 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:26.177 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:26.178 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:38:26.178 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:26.178 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:26.178 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:26.179 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:26.179 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:38:26.179 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:26.179 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:26.180 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:26.180 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:26.180 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:26.180 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:26.181 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:26.181 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:26.181 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:26.181 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:26.181 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:26.182 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:38:26.182 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:26.182 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:26.182 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:26.183 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:26.183 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:26.183 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:26.183 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:26.184 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:26.184 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:26.184 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:26.184 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:38:26.185 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:26.185 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:38:26.185 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:38:26.185 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:38:26.185 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:26.186 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:38:26.187 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:38:31.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:31.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:31.190 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:38:31.239 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:38:31.239 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:38:31.319 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:38:31.319 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:38:31.320 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:38:31.320 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:38:31.320 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:38:31.320 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:38:31.321 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:31.321 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:31.321 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:38:31.321 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:31.322 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:38:31.322 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:31.322 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:31.322 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:38:31.322 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:31.323 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:38:31.323 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:31.323 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:38:31.323 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:31.324 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:31.324 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:31.324 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:31.324 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:38:31.325 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:31.325 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:31.325 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:38:31.325 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:31.325 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:31.326 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:31.326 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:38:31.326 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:31.326 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:31.327 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:31.327 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:31.327 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:38:31.327 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:31.327 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:31.328 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:31.328 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:31.328 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:31.328 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:31.329 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:31.329 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:31.329 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:31.329 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:31.330 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:31.330 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:38:31.330 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:31.330 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:31.331 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:31.331 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:31.331 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:31.331 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:31.332 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:31.332 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:31.332 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:31.332 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:31.333 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:38:31.333 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:31.333 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:38:31.333 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:38:31.333 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:38:31.334 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:31.334 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:38:31.335 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:38:36.336 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:36.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:36.338 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:38:36.386 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:38:36.386 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:38:36.461 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:38:36.461 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:38:36.461 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:38:36.462 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:38:36.462 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:38:36.462 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:38:36.462 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:36.463 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:36.463 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:38:36.463 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:36.463 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:38:36.464 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:36.464 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:36.464 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:38:36.464 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:36.465 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:38:36.465 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:36.465 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:38:36.465 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:36.465 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:36.466 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:36.466 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:36.466 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:38:36.466 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:36.467 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:36.467 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:38:36.467 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:36.467 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:36.467 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:36.468 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:38:36.468 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:36.468 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:36.468 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:36.469 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:36.469 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:38:36.469 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:36.469 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:36.469 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:36.470 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:36.470 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:36.470 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:36.470 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:36.470 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:36.471 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:36.471 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:36.471 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:36.471 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:38:36.472 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:36.472 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:36.472 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:36.472 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:36.472 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:36.473 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:36.473 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:36.473 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:36.473 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:36.474 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:36.474 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:38:36.474 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:36.474 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:38:36.474 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:38:36.475 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:38:36.475 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:36.475 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:38:36.476 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:38:41.478 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:41.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:41.479 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:38:41.529 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:38:41.530 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:38:41.601 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:38:41.601 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:38:41.601 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:38:41.602 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:38:41.602 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:38:41.602 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:38:41.602 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:41.603 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:41.603 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:38:41.603 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:41.603 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:38:41.604 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:41.604 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:41.604 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:38:41.604 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:41.605 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:38:41.605 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:41.605 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:38:41.605 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:41.606 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:41.606 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:41.606 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:41.607 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:38:41.607 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:41.607 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:41.607 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.869904_161.1", 2021-09-02T07:38:41.607 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:41.608 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:41.608 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:41.608 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.869904Z", 2021-09-02T07:38:41.609 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:41.609 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:41.609 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:41.609 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:41.609 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:38:41.610 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:41.610 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:41.610 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:41.610 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:41.611 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:41.611 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:41.611 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:41.611 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:41.612 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:41.612 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:41.612 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:41.612 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:38:41.613 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:41.613 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:41.613 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:41.613 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:41.614 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:41.614 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:41.614 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:41.615 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:41.615 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:41.615 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:41.615 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:38:41.616 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:41.616 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:38:41.616 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:38:41.616 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:38:41.617 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:41.617 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:38:41.618 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.869904_161.1 2021-09-02T07:38:46.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:46.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:46.621 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:38:46.665 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:38:46.665 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:38:46.728 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:38:46.728 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:38:46.729 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:38:46.729 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:38:46.729 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:38:46.729 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:38:46.730 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:46.730 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:46.730 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:38:46.730 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:46.731 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:38:46.731 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:46.731 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:46.731 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:38:46.732 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:46.732 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:38:46.732 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:46.732 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:38:46.733 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:46.733 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:46.733 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:46.733 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:46.734 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:38:46.734 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:46.734 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:46.734 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.869904_161.1", 2021-09-02T07:38:46.735 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:46.735 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:46.735 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:46.735 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.869904Z", 2021-09-02T07:38:46.736 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:46.736 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:46.736 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:46.737 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:46.737 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:38:46.738 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:46.738 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:46.738 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:46.739 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:46.739 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:46.739 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:46.739 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:46.740 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:46.740 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:46.740 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:46.740 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:46.741 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:38:46.741 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:46.741 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:46.741 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:46.742 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:46.742 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:46.742 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:46.742 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:46.743 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:46.743 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:46.743 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:46.743 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:38:46.744 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:46.744 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:38:46.744 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:38:46.745 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:38:46.745 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:46.745 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:38:46.747 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.869904_161.1 2021-09-02T07:38:51.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:51.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:51.749 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:38:51.800 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:38:51.801 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:38:51.875 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:38:51.876 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:38:51.876 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:38:51.876 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:38:51.876 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:38:51.876 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:38:51.877 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:51.877 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:51.877 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:38:51.877 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:51.878 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:38:51.878 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:51.878 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:51.878 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:38:51.879 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:51.879 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:38:51.879 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:51.880 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:38:51.880 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:51.880 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:51.881 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:51.881 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:51.881 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:38:51.881 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:51.882 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:51.882 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:38:51.882 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:51.882 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:51.883 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:51.883 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:38:51.883 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:51.883 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:51.883 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:51.884 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:51.884 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:38:51.884 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:51.884 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:51.885 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:51.885 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:51.885 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:51.885 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:51.886 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:51.886 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:51.886 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:51.886 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:51.886 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:51.887 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:38:51.887 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:51.887 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:51.887 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:51.888 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:51.888 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:51.888 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:51.888 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:51.888 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:51.889 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:51.889 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:51.889 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:38:51.889 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:51.890 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:38:51.890 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:38:51.890 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:38:51.890 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:51.890 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:38:51.892 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:38:56.893 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:56.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:38:56.894 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:38:56.941 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:38:56.941 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:38:57.019 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:38:57.020 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:38:57.020 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:38:57.020 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:38:57.020 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:38:57.020 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:38:57.021 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:57.021 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:57.021 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:38:57.021 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:57.022 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:38:57.022 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:57.022 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:57.022 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:38:57.023 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:57.023 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:38:57.023 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:57.023 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:38:57.023 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:57.024 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:57.024 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:57.024 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:57.024 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:38:57.025 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:57.025 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:57.025 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:38:57.026 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:57.026 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:57.027 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:57.027 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:38:57.027 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:57.027 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:57.028 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:57.028 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:57.028 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:38:57.029 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:57.029 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:57.029 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:57.029 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:57.029 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:57.030 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:57.030 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:57.030 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:57.030 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:57.031 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:57.031 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:38:57.031 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:38:57.031 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:38:57.032 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:38:57.032 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:38:57.032 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:38:57.032 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:38:57.033 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:38:57.033 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:38:57.033 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:38:57.033 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:57.033 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:57.034 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:38:57.034 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:38:57.034 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:38:57.034 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:38:57.035 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:38:57.035 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:38:57.035 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:38:57.037 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:39:02.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:02.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:02.040 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:39:02.085 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:39:02.085 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:39:02.153 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:39:02.153 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:39:02.153 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:39:02.154 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:39:02.154 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:39:02.154 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:39:02.154 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:02.155 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:02.155 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:39:02.155 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:02.156 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:39:02.156 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:02.156 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:02.156 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:39:02.157 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:02.157 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:39:02.157 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:02.157 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:39:02.157 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:02.158 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:02.158 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:02.159 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:02.159 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:39:02.159 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:02.159 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:02.160 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T07:39:02.163 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:02.163 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:02.164 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:02.164 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T07:39:02.164 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:02.164 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:02.165 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:02.165 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:02.165 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:39:02.165 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:02.166 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:02.166 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:02.166 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:02.166 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:02.166 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:02.167 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:02.167 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:02.167 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:02.167 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:02.168 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:02.168 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:39:02.168 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:02.168 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:02.168 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:02.169 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:02.169 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:02.169 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:02.169 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:02.170 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:02.170 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:02.170 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:02.170 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:39:02.171 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:02.171 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:39:02.171 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:39:02.171 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:39:02.171 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:02.172 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:39:02.173 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.906521_163.1 2021-09-02T07:39:07.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:07.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:07.176 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:39:07.225 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:39:07.225 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:39:07.313 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:39:07.313 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:39:07.314 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:39:07.314 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:39:07.314 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:39:07.315 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:39:07.315 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:07.315 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:07.315 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:39:07.315 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:07.316 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:39:07.316 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:07.316 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:07.316 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:39:07.317 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:07.317 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:39:07.317 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:07.318 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:39:07.318 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:07.318 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:07.318 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:07.319 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:07.319 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:39:07.319 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:07.319 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:07.319 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:39:07.320 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:07.320 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:07.320 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:07.320 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:39:07.320 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:07.321 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:07.321 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:07.321 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:07.321 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:39:07.322 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:07.322 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:07.322 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:07.323 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:07.323 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:07.323 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:07.323 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:07.323 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:07.324 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:07.324 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:07.324 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:07.324 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:39:07.325 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:07.325 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:07.325 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:07.325 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:07.326 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:07.326 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:07.326 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:07.327 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:07.327 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:07.327 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:07.327 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:39:07.327 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:07.328 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:39:07.328 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:39:07.328 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:39:07.328 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:07.329 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:39:07.330 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:39:12.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:12.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:12.333 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:39:12.382 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:39:12.383 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:39:12.457 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:39:12.457 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:39:12.457 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:39:12.458 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:39:12.458 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:39:12.459 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:39:12.459 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:12.459 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:12.459 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:39:12.460 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:12.460 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:39:12.460 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:12.461 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:12.461 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:39:12.462 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:12.462 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:39:12.462 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:12.462 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:39:12.463 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:12.463 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:12.463 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:12.464 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:12.464 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:39:12.464 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:12.464 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:12.464 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T07:39:12.465 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:12.465 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:12.465 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:12.465 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T07:39:12.466 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:12.466 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:12.466 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:12.466 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:12.467 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:39:12.467 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:12.467 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:12.467 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:12.468 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:12.468 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:12.468 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:12.469 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:12.469 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:12.470 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:12.470 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:12.471 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:12.471 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:39:12.471 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:12.472 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:12.472 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:12.472 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:12.473 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:12.473 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:12.474 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:12.474 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:12.474 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:12.474 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:12.475 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:39:12.475 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:12.475 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:39:12.476 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:39:12.476 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:39:12.476 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:12.476 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:39:12.478 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.910082_164.1 2021-09-02T07:39:17.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:17.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:17.480 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:39:17.529 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:39:17.529 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:39:17.602 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:39:17.603 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:39:17.603 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:39:17.603 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:39:17.603 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:39:17.604 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:39:17.604 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:17.604 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:17.604 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:39:17.605 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:17.605 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:39:17.605 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:17.605 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:17.606 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:39:17.606 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:17.606 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:39:17.606 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:17.607 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:39:17.608 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:17.608 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:17.608 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:17.608 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:17.609 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:39:17.609 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:17.609 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:17.609 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:39:17.610 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:17.610 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:17.610 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:17.611 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:39:17.612 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:17.612 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:17.612 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:17.613 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:17.613 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:39:17.613 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:17.613 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:17.613 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:17.614 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:17.614 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:17.614 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:17.614 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:17.615 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:17.615 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:17.615 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:17.615 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:17.616 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:39:17.616 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:17.616 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:17.616 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:17.616 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:17.617 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:17.617 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:17.617 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:17.617 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:17.618 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:17.618 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:17.618 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:39:17.618 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:17.618 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:39:17.619 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:39:17.619 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:39:17.619 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:17.619 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:39:17.621 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:39:22.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:22.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:22.624 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:39:22.666 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:39:22.667 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:39:22.740 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:39:22.740 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:39:22.741 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:39:22.741 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:39:22.741 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:39:22.741 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:39:22.742 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:22.742 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:22.742 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:39:22.742 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:22.743 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:39:22.743 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:22.743 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:22.743 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:39:22.743 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:22.744 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:39:22.744 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:22.744 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:39:22.744 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:22.744 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:22.745 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:22.745 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:22.745 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:39:22.746 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:22.746 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:22.746 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.844214_158.1", 2021-09-02T07:39:22.746 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:22.747 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:22.747 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:22.747 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.844214Z", 2021-09-02T07:39:22.747 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:22.748 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:22.748 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:22.748 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:22.748 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:39:22.749 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:22.749 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:22.749 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:22.749 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:22.749 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:22.750 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:22.750 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:22.750 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:22.750 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:22.751 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:22.751 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:22.751 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:39:22.751 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:22.751 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:22.752 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:22.752 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:22.752 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:22.752 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:22.753 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:22.753 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:22.753 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:22.753 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:22.753 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:39:22.754 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:22.754 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:39:22.754 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:39:22.754 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:39:22.755 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:22.755 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:39:22.756 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.844214_158.1 2021-09-02T07:39:27.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:27.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:27.760 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:39:27.809 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:39:27.809 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:39:27.882 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:39:27.883 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:39:27.883 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:39:27.883 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:39:27.884 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:39:27.884 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:39:27.884 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:27.884 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:27.885 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:39:27.885 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:27.885 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:39:27.885 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:27.886 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:27.886 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:39:27.886 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:27.886 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:39:27.887 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:27.887 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:39:27.887 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:27.887 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:27.887 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:27.888 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:27.888 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:39:27.888 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:27.888 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:27.889 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T07:39:27.889 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:27.889 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:27.889 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:27.890 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T07:39:27.890 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:27.890 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:27.890 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:27.891 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:27.891 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:39:27.891 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:27.891 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:27.892 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:27.892 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:27.892 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:27.892 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:27.892 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:27.893 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:27.893 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:27.893 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:27.893 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:27.894 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:39:27.894 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:27.894 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:27.895 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:27.895 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:27.895 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:27.895 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:27.896 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:27.896 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:27.896 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:27.896 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:27.897 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:39:27.897 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:27.897 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:39:27.897 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:39:27.898 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:39:27.898 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:27.898 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:39:27.900 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.910082_164.1 2021-09-02T07:39:32.902 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:32.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:32.903 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:39:32.952 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:39:32.952 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:39:33.028 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:39:33.028 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:39:33.028 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:39:33.029 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:39:33.029 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:39:33.029 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:39:33.029 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:33.030 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:33.030 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:39:33.030 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:33.030 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:39:33.030 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:33.031 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:33.031 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:39:33.031 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:33.031 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:39:33.032 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:33.032 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:39:33.032 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:33.032 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:33.032 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:33.033 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:33.033 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:39:33.033 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:33.033 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:33.034 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:39:33.034 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:33.034 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:33.034 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:33.035 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:39:33.035 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:33.035 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:33.035 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:33.036 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:33.036 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:39:33.036 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:33.036 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:33.037 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:33.037 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:33.037 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:33.037 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:33.037 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:33.038 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:33.038 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:33.038 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:33.038 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:33.039 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:39:33.039 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:33.039 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:33.039 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:33.039 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:33.040 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:33.040 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:33.040 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:33.040 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:33.041 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:33.041 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:33.041 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:39:33.041 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:33.041 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:39:33.042 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:39:33.042 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:39:33.042 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:33.042 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:39:33.044 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:39:38.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:38.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:38.048 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:39:38.093 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:39:38.093 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:39:38.170 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:39:38.171 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:39:38.171 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:39:38.171 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:39:38.171 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:39:38.172 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:39:38.172 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:38.172 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:38.172 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:39:38.173 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:38.173 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:39:38.173 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:38.174 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:38.174 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:39:38.174 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:38.174 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:39:38.175 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:38.175 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:39:38.175 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:38.175 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:38.175 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:38.176 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:38.176 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:39:38.176 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:38.176 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:38.177 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.985199_169.1", 2021-09-02T07:39:38.177 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:38.177 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:38.177 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:38.177 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.985199Z", 2021-09-02T07:39:38.178 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:38.178 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:38.178 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:38.178 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:38.179 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:39:38.179 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:38.179 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:38.179 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:38.180 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:38.180 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:38.180 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:38.180 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:38.180 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:38.181 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:38.181 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:38.181 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:38.181 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:39:38.182 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:38.182 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:38.182 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:38.182 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:38.182 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:38.183 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:38.183 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:38.183 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:38.183 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:38.183 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:38.184 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:39:38.184 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:38.184 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:39:38.184 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:39:38.185 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:39:38.185 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:38.185 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:39:38.187 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.985199_169.1 2021-09-02T07:39:43.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:43.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:43.189 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:39:43.234 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:39:43.235 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:39:43.299 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:39:43.299 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:39:43.300 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:39:43.300 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:39:43.300 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:39:43.300 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:39:43.301 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:43.301 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:43.301 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:39:43.301 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:43.302 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:39:43.302 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:43.302 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:43.302 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:39:43.303 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:43.303 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:39:43.303 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:43.303 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:39:43.303 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:43.304 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:43.304 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:43.305 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:43.305 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:39:43.305 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:43.305 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:43.306 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:39:43.306 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:43.306 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:43.306 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:43.307 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:39:43.307 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:43.307 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:43.307 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:43.307 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:43.308 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:39:43.308 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:43.308 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:43.308 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:43.309 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:43.309 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:43.309 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:43.309 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:43.309 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:43.310 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:43.310 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:43.310 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:43.310 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:39:43.311 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:43.311 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:43.311 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:43.311 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:43.311 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:43.313 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:43.313 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:43.314 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:43.314 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:43.314 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:43.315 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:39:43.315 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:43.315 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:39:43.316 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:39:43.316 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:39:43.316 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:43.317 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:39:43.318 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:39:48.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:48.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:48.321 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:39:48.370 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:39:48.370 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:39:48.446 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:39:48.446 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:39:48.447 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:39:48.447 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:39:48.447 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:39:48.447 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:39:48.447 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:48.448 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:48.448 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:39:48.448 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:48.448 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:39:48.449 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:48.449 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:48.449 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:39:48.449 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:48.450 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:39:48.450 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:48.450 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:39:48.450 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:48.451 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:48.451 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:48.451 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:48.451 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:39:48.452 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:48.452 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:48.452 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.099221_173.1", 2021-09-02T07:39:48.452 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:48.452 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:48.453 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:48.453 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.099221Z", 2021-09-02T07:39:48.453 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:48.453 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:48.454 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:48.454 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:48.454 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:39:48.454 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:48.455 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:48.455 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:48.455 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:48.455 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:48.455 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:48.456 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:48.456 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:48.456 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:48.456 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:48.456 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:48.457 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:39:48.457 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:48.457 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:48.457 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:48.458 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:48.458 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:48.458 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:48.458 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:48.458 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:48.459 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:48.459 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:48.459 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:39:48.459 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:48.460 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:39:48.460 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:39:48.460 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:39:48.460 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:48.460 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:39:48.462 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568127.099221_173.1 2021-09-02T07:39:53.463 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:53.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:53.464 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:39:53.509 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:39:53.509 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:39:53.584 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:39:53.584 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:39:53.584 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:39:53.584 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:39:53.585 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:39:53.585 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:39:53.585 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:53.585 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:53.586 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:39:53.586 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:53.586 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:39:53.586 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:53.587 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:53.587 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:39:53.587 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:53.587 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:39:53.587 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:53.588 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:39:53.588 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:53.588 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:53.588 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:53.589 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:53.589 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:39:53.589 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:53.589 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:53.589 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.932719_168.1", 2021-09-02T07:39:53.590 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:53.590 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:53.590 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:53.590 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.932719Z", 2021-09-02T07:39:53.591 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:53.591 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:53.591 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:53.591 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:53.591 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:39:53.592 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:53.592 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:53.592 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:53.592 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:53.593 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:53.593 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:53.593 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:53.593 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:53.593 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:53.594 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:53.594 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:53.594 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:39:53.594 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:53.595 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:53.595 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:53.595 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:53.595 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:53.595 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:53.596 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:53.596 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:53.596 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:53.596 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:53.597 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:39:53.597 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:53.597 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:39:53.597 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:39:53.597 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:39:53.598 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:53.598 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:39:53.599 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.932719_168.1 2021-09-02T07:39:58.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:58.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:39:58.602 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:39:58.647 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:39:58.647 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:39:58.720 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:39:58.720 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:39:58.721 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:39:58.721 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:39:58.721 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:39:58.721 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:39:58.722 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:58.722 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:58.722 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:39:58.722 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:58.723 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:39:58.723 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:58.723 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:58.723 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:39:58.723 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:58.724 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:39:58.724 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:58.724 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:39:58.724 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:58.724 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:58.725 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:58.725 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:58.725 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:39:58.725 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:58.726 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:58.726 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:39:58.726 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:58.726 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:58.726 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:58.727 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:39:58.727 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:58.727 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:58.727 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:58.728 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:58.728 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:39:58.728 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:58.728 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:58.728 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:58.729 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:58.729 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:58.729 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:58.729 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:58.730 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:58.730 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:58.730 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:58.730 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:39:58.730 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:39:58.731 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:39:58.731 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:39:58.731 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:39:58.731 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:39:58.732 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:39:58.732 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:39:58.732 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:39:58.732 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:39:58.732 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:58.733 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:58.733 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:39:58.733 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:39:58.733 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:39:58.733 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:39:58.734 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:39:58.734 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:39:58.734 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:39:58.735 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:40:03.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:03.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:03.738 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:40:03.788 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:40:03.788 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:40:03.857 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:40:03.857 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:40:03.857 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:40:03.857 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:40:03.858 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:40:03.858 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:40:03.858 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:03.858 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:03.859 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:40:03.859 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:03.859 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:40:03.859 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:03.860 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:03.860 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:40:03.860 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:03.860 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:40:03.861 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:03.861 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:40:03.861 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:03.861 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:03.861 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:03.862 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:03.862 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:40:03.862 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:03.862 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:03.863 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:40:03.863 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:03.863 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:03.863 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:03.863 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:40:03.864 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:03.864 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:03.864 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:03.864 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:03.864 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:40:03.865 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:03.865 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:03.865 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:03.865 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:03.866 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:03.866 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:03.866 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:03.866 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:03.866 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:03.867 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:03.867 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:03.867 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:40:03.867 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:03.867 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:03.868 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:03.868 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:03.868 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:03.868 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:03.869 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:03.869 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:03.869 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:03.869 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:03.869 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:40:03.870 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:03.870 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:40:03.870 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:40:03.870 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:40:03.871 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:03.871 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:40:03.872 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:40:08.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:08.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:08.875 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:40:08.923 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:40:08.924 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:40:08.995 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:40:08.996 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:40:08.996 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:40:08.996 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:40:08.996 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:40:08.997 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:40:08.997 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:08.997 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:08.997 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:40:08.997 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:08.998 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:40:08.998 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:08.998 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:08.998 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:40:08.999 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:08.999 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:40:08.999 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:08.999 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:40:09.000 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:09.000 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:09.000 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:09.000 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:09.000 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:40:09.001 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:09.001 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:09.001 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:40:09.001 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:09.002 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:09.002 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:09.002 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:40:09.002 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:09.003 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:09.003 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:09.004 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:09.004 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:40:09.004 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:09.004 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:09.004 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:09.005 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:09.005 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:09.005 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:09.005 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:09.006 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:09.006 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:09.006 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:09.006 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:09.007 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:40:09.007 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:09.007 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:09.007 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:09.008 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:09.008 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:09.008 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:09.008 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:09.009 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:09.009 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:09.010 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:09.010 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:40:09.011 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:09.011 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:40:09.011 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:40:09.011 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:40:09.012 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:09.012 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:40:09.013 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:40:14.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:14.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:14.016 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:40:14.065 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:40:14.066 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:40:14.143 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:40:14.144 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:40:14.144 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:40:14.144 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:40:14.144 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:40:14.145 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:40:14.145 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:14.145 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:14.145 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:40:14.146 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:14.146 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:40:14.147 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:14.147 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:14.147 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:40:14.147 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:14.148 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:40:14.148 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:14.148 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:40:14.148 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:14.148 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:14.149 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:14.149 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:14.149 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:40:14.149 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:14.150 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:14.150 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T07:40:14.150 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:14.150 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:14.151 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:14.151 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T07:40:14.151 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:14.151 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:14.151 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:14.152 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:14.152 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:40:14.152 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:14.152 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:14.153 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:14.153 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:14.153 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:14.153 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:14.153 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:14.154 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:14.154 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:14.154 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:14.154 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:14.155 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:40:14.155 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:14.155 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:14.155 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:14.155 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:14.156 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:14.156 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:14.156 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:14.156 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:14.157 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:14.157 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:14.157 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:40:14.157 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:14.157 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:40:14.158 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:40:14.158 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:40:14.158 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:14.158 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:40:14.160 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.875533_162.1 2021-09-02T07:40:19.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:19.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:19.162 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:40:19.210 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:40:19.210 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:40:19.274 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:40:19.274 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:40:19.275 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:40:19.275 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:40:19.275 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:40:19.275 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:40:19.276 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:19.276 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:19.276 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:40:19.276 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:19.277 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:40:19.277 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:19.277 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:19.277 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:40:19.278 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:19.278 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:40:19.278 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:19.278 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:40:19.278 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:19.279 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:19.279 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:19.279 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:19.279 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:40:19.280 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:19.280 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:19.280 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.835437_156.1", 2021-09-02T07:40:19.280 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:19.281 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:19.281 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:19.281 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.835437Z", 2021-09-02T07:40:19.281 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:19.281 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:19.282 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:19.282 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:19.282 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:40:19.282 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:19.282 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:19.283 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:19.283 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:19.283 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:19.284 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:19.284 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:19.284 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:19.284 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:19.285 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:19.285 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:19.285 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:40:19.285 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:19.286 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:19.286 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:19.286 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:19.286 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:19.287 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:19.287 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:19.287 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:19.287 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:19.287 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:19.288 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:40:19.288 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:19.288 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:40:19.288 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:40:19.289 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:40:19.289 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:19.289 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:40:19.290 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.835437_156.1 2021-09-02T07:40:24.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:24.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:24.293 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:40:24.343 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:40:24.343 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:40:24.426 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:40:24.426 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:40:24.426 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:40:24.427 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:40:24.427 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:40:24.427 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:40:24.427 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:24.428 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:24.428 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:40:24.428 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:24.428 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:40:24.429 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:24.429 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:24.429 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:40:24.429 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:24.429 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:40:24.430 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:24.430 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:40:24.430 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:24.430 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:24.431 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:24.431 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:24.431 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:40:24.431 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:24.432 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:24.432 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:40:24.432 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:24.432 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:24.432 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:24.433 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:40:24.433 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:24.433 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:24.433 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:24.434 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:24.434 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:40:24.434 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:24.434 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:24.435 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:24.435 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:24.435 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:24.435 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:24.436 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:24.436 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:24.436 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:24.436 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:24.436 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:24.437 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:40:24.437 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:24.437 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:24.437 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:24.438 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:24.438 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:24.438 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:24.438 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:24.439 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:24.439 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:24.439 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:24.439 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:40:24.439 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:24.440 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:40:24.440 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:40:24.440 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:40:24.440 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:24.441 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:40:24.442 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:40:29.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:29.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:29.444 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:40:29.493 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:40:29.494 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:40:29.567 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:40:29.568 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:40:29.568 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:40:29.568 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:40:29.568 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:40:29.569 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:40:29.569 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:29.569 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:29.569 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:40:29.570 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:29.570 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:40:29.570 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:29.570 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:29.571 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:40:29.571 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:29.571 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:40:29.571 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:29.571 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:40:29.572 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:29.572 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:29.572 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:29.572 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:29.573 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:40:29.573 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:29.573 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:29.573 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:40:29.573 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:29.574 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:29.574 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:29.574 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:40:29.574 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:29.575 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:29.576 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:29.576 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:29.577 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:40:29.577 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:29.577 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:29.577 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:29.577 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:29.578 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:29.578 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:29.578 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:29.578 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:29.579 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:29.579 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:29.579 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:29.579 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:40:29.580 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:29.580 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:29.580 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:29.580 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:29.580 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:29.581 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:29.581 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:29.581 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:29.581 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:29.582 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:29.582 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:40:29.582 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:29.582 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:40:29.583 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:40:29.583 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:40:29.583 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:29.583 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:40:29.585 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:40:34.586 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:34.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:34.588 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:40:34.636 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:40:34.636 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:40:34.720 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:40:34.720 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:40:34.721 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:40:34.721 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:40:34.721 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:40:34.721 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:40:34.722 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:34.722 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:34.722 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:40:34.722 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:34.723 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:40:34.723 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:34.723 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:34.723 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:40:34.723 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:34.724 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:40:34.724 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:34.724 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:40:34.724 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:34.725 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:34.725 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:34.725 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:34.725 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:40:34.725 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:34.726 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:34.726 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:40:34.726 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:34.726 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:34.727 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:34.727 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:40:34.727 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:34.727 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:34.727 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:34.728 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:34.728 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:40:34.728 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:34.728 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:34.729 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:34.729 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:34.729 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:34.729 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:34.730 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:34.730 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:34.730 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:34.730 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:34.731 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:34.731 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:40:34.731 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:34.731 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:34.732 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:34.732 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:34.732 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:34.732 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:34.733 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:34.733 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:34.733 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:34.733 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:34.734 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:40:34.734 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:34.734 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:40:34.734 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:40:34.734 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:40:34.735 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:34.735 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:40:34.736 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:40:39.738 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:39.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:39.740 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:40:40.529 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:40:40.529 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:40:40.920 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:40:40.920 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:40:40.921 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:40:40.921 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:40:40.921 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:40:40.921 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:40:40.922 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:40.922 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:40.922 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:40:40.922 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:40.923 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:40:40.923 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:40.923 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:40.923 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:40:40.923 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:40.924 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:40:40.924 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:40.924 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:40:40.925 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:40.925 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:40.925 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:40.925 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:40.926 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:40:40.926 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:40.926 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:40.927 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:40:40.927 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:40.927 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:40.927 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:40.928 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:40:40.928 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:40.928 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:40.928 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:40.928 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:40.929 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:40:40.929 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:40.929 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:40.929 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:40.930 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:40.930 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:40.930 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:40.930 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:40.931 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:40.931 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:40.931 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:40.932 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:40.932 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:40:40.932 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:40.932 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:40.933 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:40.933 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:40.933 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:40.933 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:40.934 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:40.934 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:40.934 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:40.935 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:40.935 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:40:40.936 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:40.936 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:40:40.937 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:40:40.937 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:40:40.937 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:40.937 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:40:40.943 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:40:45.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:45.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:45.946 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:40:46.058 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:40:46.058 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:40:46.303 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:40:46.303 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:40:46.304 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:40:46.304 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:40:46.304 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:40:46.304 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:40:46.304 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:46.305 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:46.305 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:40:46.305 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:46.305 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:40:46.306 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:46.306 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:46.306 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:40:46.306 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:46.306 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:40:46.307 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:46.307 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:40:46.307 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:46.307 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:46.308 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:46.308 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:46.308 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:40:46.308 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:46.308 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:46.309 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:40:46.309 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:46.309 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:46.309 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:46.309 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:40:46.310 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:46.310 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:46.310 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:46.310 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:46.311 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:40:46.311 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:46.311 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:46.311 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:46.311 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:46.312 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:46.312 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:46.312 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:46.312 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:46.313 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:46.313 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:46.313 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:46.313 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:40:46.313 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:46.314 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:46.314 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:46.314 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:46.314 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:46.314 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:46.315 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:46.315 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:46.315 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:46.315 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:46.316 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:40:46.316 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:46.316 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:40:46.316 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:40:46.316 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:40:46.317 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:46.317 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:40:46.318 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:40:51.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:51.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:51.320 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:40:51.465 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:40:51.465 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:40:51.559 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:40:51.559 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:40:51.559 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:40:51.560 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:40:51.560 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:40:51.560 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:40:51.560 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:51.560 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:51.561 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:40:51.561 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:51.561 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:40:51.561 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:51.562 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:51.562 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:40:51.562 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:51.562 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:40:51.563 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:51.563 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:40:51.563 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:51.563 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:51.563 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:51.564 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:51.564 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:40:51.564 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:51.565 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:51.565 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.929348_167.1", 2021-09-02T07:40:51.565 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:51.565 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:51.565 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:51.566 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.929348Z", 2021-09-02T07:40:51.566 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:51.566 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:51.566 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:51.567 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:51.567 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:40:51.567 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:51.568 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:51.568 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:51.568 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:51.568 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:51.569 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:51.569 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:51.569 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:51.569 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:51.569 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:51.570 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:51.570 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:40:51.570 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:51.570 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:51.571 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:51.571 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:51.571 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:51.571 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:51.572 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:51.572 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:51.572 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:51.572 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:51.572 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:40:51.573 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:51.573 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:40:51.573 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:40:51.573 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:40:51.574 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:51.574 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:40:51.575 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.929348_167.1 2021-09-02T07:40:56.577 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:56.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:40:56.578 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:40:56.655 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:40:56.655 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:40:56.827 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:40:56.827 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:40:56.828 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:40:56.828 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:40:56.828 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:40:56.829 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:40:56.829 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:56.829 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:56.829 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:40:56.830 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:56.830 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:40:56.830 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:56.830 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:56.831 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:40:56.831 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:56.831 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:40:56.831 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:56.831 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:40:56.832 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:56.832 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:56.832 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:56.832 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:56.833 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:40:56.833 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:56.833 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:56.833 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.835437_156.1", 2021-09-02T07:40:56.834 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:56.834 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:56.834 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:56.834 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.835437Z", 2021-09-02T07:40:56.834 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:56.835 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:56.835 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:56.835 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:56.836 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:40:56.836 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:56.836 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:56.837 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:56.837 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:56.837 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:56.837 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:56.838 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:56.838 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:56.838 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:56.838 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:56.839 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:40:56.839 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:40:56.839 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:40:56.839 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:40:56.840 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:40:56.840 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:40:56.840 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:40:56.840 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:40:56.841 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:40:56.841 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:40:56.841 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:56.841 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:56.841 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:40:56.842 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:40:56.842 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:40:56.842 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:40:56.842 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:40:56.843 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:40:56.843 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:40:56.844 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.835437_156.1 2021-09-02T07:41:01.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:01.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:01.848 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:41:01.890 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:41:01.890 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:41:01.968 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:41:01.969 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:41:01.969 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:41:01.969 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:41:01.970 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:41:01.970 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:41:01.970 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:01.970 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:01.971 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:41:01.971 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:01.971 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:41:01.971 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:01.972 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:01.972 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:41:01.972 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:01.972 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:41:01.972 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:01.973 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:41:01.973 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:01.973 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:01.973 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:01.974 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:01.974 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:41:01.974 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:01.974 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:01.974 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:41:01.975 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:01.975 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:01.975 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:01.975 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:41:01.976 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:01.976 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:01.976 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:01.976 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:01.976 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:41:01.977 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:01.977 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:01.977 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:01.977 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:01.978 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:01.978 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:01.978 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:01.978 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:01.979 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:01.979 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:01.979 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:01.979 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:41:01.979 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:01.980 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:01.980 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:01.980 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:01.980 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:01.981 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:01.981 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:01.981 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:01.981 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:01.981 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:01.982 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:41:01.982 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:01.982 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:41:01.982 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:41:01.983 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:41:01.983 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:01.983 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:41:01.984 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:41:06.986 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:06.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:06.988 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:41:07.049 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:41:07.050 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:41:07.121 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:41:07.121 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:41:07.121 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:41:07.122 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:41:07.122 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:41:07.122 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:41:07.122 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:07.123 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:07.123 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:41:07.123 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:07.123 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:41:07.123 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:07.124 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:07.124 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:41:07.124 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:07.124 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:41:07.125 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:07.125 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:41:07.125 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:07.125 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:07.125 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:07.126 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:07.126 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:41:07.126 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:07.126 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:07.127 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:41:07.127 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:07.127 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:07.127 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:07.127 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:41:07.128 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:07.128 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:07.128 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:07.128 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:07.128 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:41:07.129 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:07.129 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:07.129 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:07.129 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:07.130 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:07.130 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:07.130 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:07.130 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:07.130 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:07.131 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:07.131 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:07.131 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:41:07.132 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:07.132 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:07.132 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:07.132 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:07.133 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:07.133 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:07.133 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:07.133 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:07.133 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:07.134 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:07.134 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:41:07.134 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:07.134 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:41:07.135 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:41:07.135 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:41:07.135 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:07.135 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:41:07.137 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:41:12.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:12.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:12.140 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:41:12.182 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:41:12.183 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:41:12.258 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:41:12.259 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:41:12.259 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:41:12.260 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:41:12.260 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:41:12.260 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:41:12.261 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:12.261 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:12.261 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:41:12.262 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:12.262 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:41:12.262 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:12.262 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:12.262 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:41:12.263 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:12.263 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:41:12.263 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:12.263 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:41:12.264 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:12.264 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:12.264 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:12.264 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:12.264 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:41:12.265 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:12.265 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:12.265 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.043408_172.1", 2021-09-02T07:41:12.265 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:12.266 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:12.266 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:12.267 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.043408Z", 2021-09-02T07:41:12.267 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:12.267 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:12.268 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:12.268 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:12.268 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:41:12.268 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:12.269 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:12.269 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:12.269 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:12.270 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:12.270 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:12.270 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:12.270 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:12.271 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:12.271 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:12.271 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:12.271 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:41:12.271 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:12.272 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:12.272 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:12.273 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:12.273 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:12.273 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:12.273 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:12.274 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:12.275 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:12.275 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:12.275 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:41:12.276 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:12.276 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:41:12.277 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:41:12.277 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:41:12.277 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:12.278 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:41:12.279 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568127.043408_172.1 2021-09-02T07:41:17.281 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:17.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:17.283 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:41:17.326 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:41:17.327 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:41:17.395 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:41:17.396 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:41:17.396 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:41:17.396 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:41:17.396 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:41:17.397 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:41:17.397 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:17.397 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:17.397 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:41:17.398 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:17.398 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:41:17.398 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:17.399 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:17.399 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:41:17.399 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:17.399 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:41:17.399 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:17.400 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:41:17.400 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:17.400 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:17.400 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:17.401 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:17.401 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:41:17.401 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:17.401 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:17.401 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.911309_165.1", 2021-09-02T07:41:17.402 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:17.402 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:17.402 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:17.402 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.911309Z", 2021-09-02T07:41:17.403 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:17.403 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:17.403 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:17.403 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:17.404 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:41:17.404 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:17.404 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:17.404 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:17.404 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:17.405 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:17.405 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:17.405 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:17.405 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:17.406 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:17.406 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:17.406 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:17.406 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:41:17.407 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:17.407 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:17.407 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:17.407 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:17.408 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:17.408 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:17.408 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:17.408 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:17.409 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:17.409 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:17.409 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:41:17.409 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:17.409 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:41:17.410 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:41:17.410 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:41:17.410 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:17.410 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:41:17.412 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.911309_165.1 2021-09-02T07:41:22.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:22.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:22.415 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:41:22.459 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:41:22.459 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:41:22.522 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:41:22.523 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:41:22.523 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:41:22.523 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:41:22.523 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:41:22.524 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:41:22.524 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:22.525 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:22.525 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:41:22.525 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:22.525 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:41:22.525 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:22.526 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:22.526 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:41:22.526 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:22.526 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:41:22.527 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:22.527 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:41:22.527 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:22.527 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:22.528 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:22.528 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:22.528 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:41:22.528 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:22.529 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:22.529 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:41:22.529 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:22.529 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:22.529 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:22.530 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:41:22.530 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:22.530 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:22.530 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:22.531 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:22.531 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:41:22.531 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:22.531 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:22.532 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:22.532 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:22.532 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:22.533 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:22.533 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:22.533 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:22.533 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:22.534 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:22.534 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:22.534 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:41:22.534 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:22.535 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:22.535 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:22.535 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:22.535 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:22.536 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:22.536 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:22.536 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:22.536 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:22.536 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:22.537 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:41:22.537 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:22.537 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:41:22.537 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:41:22.538 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:41:22.538 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:22.538 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:41:22.539 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:41:27.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:27.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:27.543 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:41:27.615 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:41:27.615 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:41:27.725 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:41:27.726 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:41:27.726 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:41:27.726 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:41:27.727 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:41:27.727 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:41:27.727 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:27.727 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:27.728 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:41:27.728 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:27.728 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:41:27.728 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:27.729 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:27.729 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:41:27.729 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:27.729 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:41:27.730 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:27.730 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:41:27.730 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:27.730 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:27.731 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:27.731 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:27.731 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:41:27.731 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:27.731 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:27.732 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.835437_156.1", 2021-09-02T07:41:27.732 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:27.732 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:27.732 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:27.733 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.835437Z", 2021-09-02T07:41:27.733 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:27.733 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:27.733 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:27.734 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:27.734 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:41:27.734 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:27.734 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:27.734 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:27.735 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:27.735 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:27.735 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:27.735 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:27.736 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:27.736 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:27.736 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:27.736 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:27.737 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:41:27.737 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:27.737 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:27.737 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:27.737 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:27.738 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:27.738 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:27.738 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:27.738 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:27.739 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:27.739 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:27.739 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:41:27.739 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:27.740 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:41:27.740 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:41:27.740 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:41:27.740 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:27.740 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:41:27.742 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.835437_156.1 2021-09-02T07:41:32.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:32.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:32.744 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:41:32.816 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:41:32.817 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:41:32.946 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:41:32.946 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:41:32.947 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:41:32.947 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:41:32.947 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:41:32.947 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:41:32.948 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:32.948 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:32.948 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:41:32.948 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:32.948 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:41:32.949 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:32.949 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:32.949 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:41:32.949 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:32.950 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:41:32.950 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:32.950 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:41:32.950 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:32.950 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:32.951 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:32.951 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:32.951 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:41:32.951 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:32.952 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:32.952 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.869904_161.1", 2021-09-02T07:41:32.952 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:32.952 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:32.952 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:32.953 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.869904Z", 2021-09-02T07:41:32.953 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:32.953 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:32.953 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:32.954 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:32.954 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:41:32.954 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:32.954 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:32.955 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:32.955 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:32.955 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:32.955 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:32.956 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:32.956 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:32.956 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:32.956 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:32.957 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:32.957 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:41:32.957 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:32.957 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:32.958 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:32.958 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:32.958 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:32.958 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:32.958 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:32.959 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:32.959 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:32.959 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:32.959 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:41:32.960 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:32.960 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:41:32.960 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:41:32.960 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:41:32.960 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:32.961 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:41:32.962 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.869904_161.1 2021-09-02T07:41:37.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:37.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:37.965 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:41:38.008 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:41:38.008 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:41:38.072 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:41:38.073 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:41:38.073 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:41:38.073 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:41:38.074 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:41:38.074 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:41:38.074 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:38.074 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:38.075 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:41:38.075 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:38.075 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:41:38.075 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:38.076 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:38.076 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:41:38.076 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:38.076 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:41:38.077 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:38.077 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:41:38.077 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:38.077 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:38.077 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:38.078 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:38.078 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:41:38.078 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:38.078 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:38.079 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.869904_161.1", 2021-09-02T07:41:38.079 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:38.079 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:38.079 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:38.080 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.869904Z", 2021-09-02T07:41:38.080 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:38.080 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:38.080 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:38.081 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:38.081 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:41:38.081 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:38.081 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:38.082 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:38.082 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:38.082 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:38.082 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:38.082 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:38.083 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:38.083 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:38.083 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:38.083 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:38.084 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:41:38.084 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:38.084 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:38.084 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:38.085 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:38.085 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:38.085 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:38.085 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:38.086 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:38.086 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:38.086 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:38.086 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:41:38.087 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:38.087 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:41:38.087 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:41:38.087 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:41:38.088 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:38.088 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:41:38.089 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.869904_161.1 2021-09-02T07:41:43.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:43.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:43.093 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:41:43.151 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:41:43.152 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:41:43.220 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:41:43.220 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:41:43.220 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:41:43.220 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:41:43.221 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:41:43.221 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:41:43.221 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:43.221 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:43.222 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:41:43.222 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:43.222 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:41:43.222 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:43.223 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:43.223 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:41:43.223 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:43.223 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:41:43.223 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:43.224 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:41:43.224 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:43.224 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:43.224 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:43.225 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:43.225 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:41:43.225 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:43.225 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:43.225 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:41:43.226 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:43.226 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:43.226 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:43.226 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:41:43.227 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:43.227 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:43.227 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:43.227 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:43.227 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:41:43.228 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:43.228 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:43.228 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:43.228 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:43.229 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:43.229 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:43.229 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:43.229 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:43.230 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:43.230 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:43.230 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:43.230 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:41:43.231 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:43.231 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:43.231 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:43.231 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:43.231 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:43.232 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:43.232 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:43.232 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:43.232 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:43.233 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:43.233 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:41:43.233 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:43.233 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:41:43.233 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:41:43.234 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:41:43.234 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:43.234 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:41:43.235 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:41:48.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:48.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:48.239 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:41:48.282 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:41:48.282 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:41:48.344 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:41:48.345 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:41:48.345 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:41:48.345 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:41:48.346 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:41:48.346 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:41:48.346 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:48.346 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:48.347 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:41:48.347 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:48.347 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:41:48.347 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:48.347 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:48.348 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:41:48.348 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:48.348 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:41:48.349 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:48.349 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:41:48.349 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:48.349 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:48.349 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:48.350 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:48.350 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:41:48.350 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:48.350 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:48.351 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:41:48.351 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:48.351 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:48.351 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:48.352 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:41:48.352 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:48.352 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:48.352 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:48.353 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:48.353 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:41:48.353 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:48.353 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:48.353 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:48.354 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:48.354 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:48.354 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:48.354 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:48.355 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:48.355 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:48.355 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:48.355 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:48.356 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:41:48.356 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:48.356 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:48.356 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:48.356 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:48.357 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:48.357 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:48.357 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:48.357 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:48.357 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:48.358 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:48.358 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:41:48.358 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:48.358 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:41:48.359 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:41:48.359 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:41:48.359 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:48.359 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:41:48.360 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:41:53.362 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:53.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:53.363 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:41:53.406 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:41:53.406 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:41:53.494 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:41:53.495 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:41:53.495 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:41:53.495 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:41:53.496 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:41:53.496 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:41:53.496 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:53.496 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:53.497 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:41:53.497 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:53.497 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:41:53.497 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:53.497 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:53.498 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:41:53.498 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:53.498 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:41:53.498 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:53.499 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:41:53.499 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:53.499 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:53.500 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:53.500 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:53.500 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:41:53.500 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:53.501 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:53.501 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:41:53.501 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:53.501 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:53.501 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:53.502 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:41:53.502 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:53.502 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:53.502 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:53.503 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:53.503 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:41:53.503 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:53.503 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:53.505 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:53.505 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:53.505 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:53.506 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:53.506 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:53.506 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:53.506 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:53.507 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:53.507 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:53.507 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:41:53.507 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:53.507 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:53.508 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:53.508 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:53.508 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:53.508 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:53.509 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:53.509 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:53.509 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:53.509 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:53.510 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:41:53.510 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:53.510 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:41:53.510 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:41:53.511 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:41:53.511 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:53.511 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:41:53.511 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:41:58.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:58.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:41:58.514 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:41:58.559 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:41:58.561 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:41:58.628 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:41:58.629 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:41:58.629 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:41:58.629 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:41:58.630 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:41:58.630 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:41:58.630 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:58.630 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:58.631 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:41:58.631 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:58.631 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:41:58.631 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:58.632 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:58.632 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:41:58.632 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:58.632 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:41:58.633 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:58.633 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:41:58.633 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:58.633 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:58.634 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:58.634 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:58.635 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:41:58.635 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:58.636 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:58.636 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:41:58.636 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:58.636 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:58.637 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:58.637 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:41:58.637 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:58.637 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:58.637 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:58.638 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:58.638 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:41:58.638 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:58.638 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:58.639 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:58.639 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:58.639 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:58.639 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:58.640 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:58.640 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:58.640 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:58.640 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:58.641 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:41:58.641 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:41:58.641 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:41:58.641 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:41:58.641 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:41:58.642 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:41:58.642 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:41:58.642 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:41:58.643 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:41:58.643 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:41:58.643 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:58.643 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:58.644 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:41:58.644 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:41:58.644 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:41:58.644 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:41:58.645 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:41:58.645 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:41:58.645 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:41:58.647 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:42:03.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:03.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:03.651 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:42:03.696 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:42:03.696 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:42:03.763 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:42:03.764 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:42:03.764 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:42:03.764 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:42:03.765 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:42:03.765 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:42:03.765 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:03.765 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:03.766 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:42:03.766 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:03.766 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:42:03.766 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:03.766 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:03.767 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:42:03.767 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:03.767 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:42:03.768 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:03.768 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:42:03.768 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:03.768 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:03.768 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:03.769 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:03.769 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:42:03.769 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:03.769 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:03.770 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:42:03.770 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:03.770 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:03.770 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:03.771 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:42:03.771 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:03.771 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:03.771 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:03.772 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:03.772 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:42:03.772 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:03.772 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:03.773 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:03.773 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:03.773 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:03.773 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:03.773 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:03.774 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:03.774 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:03.774 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:03.774 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:03.775 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:42:03.775 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:03.775 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:03.775 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:03.776 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:03.776 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:03.776 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:03.776 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:03.777 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:03.777 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:03.777 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:03.777 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:42:03.777 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:03.778 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:42:03.778 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:42:03.778 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:42:03.778 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:03.778 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:42:03.780 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:42:08.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:08.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:08.784 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:42:08.829 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:42:08.829 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:42:08.901 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:42:08.901 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:42:08.902 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:42:08.902 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:42:08.902 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:42:08.902 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:42:08.903 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:08.903 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:08.903 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:42:08.903 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:08.904 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:42:08.904 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:08.905 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:08.905 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:42:08.905 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:08.905 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:42:08.906 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:08.906 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:42:08.906 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:08.906 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:08.906 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:08.907 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:08.907 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:42:08.907 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:08.907 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:08.908 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.835437_156.1", 2021-09-02T07:42:08.908 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:08.908 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:08.908 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:08.909 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.835437Z", 2021-09-02T07:42:08.909 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:08.909 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:08.909 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:08.909 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:08.910 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:42:08.910 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:08.910 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:08.910 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:08.911 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:08.911 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:08.911 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:08.911 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:08.912 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:08.912 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:08.912 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:08.912 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:08.913 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:42:08.913 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:08.913 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:08.913 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:08.913 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:08.914 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:08.914 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:08.914 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:08.914 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:08.915 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:08.915 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:08.915 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:42:08.916 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:08.916 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:42:08.916 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:42:08.916 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:42:08.916 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:08.917 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:42:08.918 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.835437_156.1 2021-09-02T07:42:13.920 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:13.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:13.921 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:42:13.963 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:42:13.964 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:42:14.027 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:42:14.028 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:42:14.028 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:42:14.028 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:42:14.028 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:42:14.029 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:42:14.029 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:14.029 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:14.029 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:42:14.030 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:14.030 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:42:14.030 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:14.030 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:14.031 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:42:14.031 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:14.031 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:42:14.031 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:14.031 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:42:14.032 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:14.032 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:14.032 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:14.032 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:14.033 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:42:14.033 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:14.033 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:14.033 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.867814_160.1", 2021-09-02T07:42:14.034 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:14.034 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:14.034 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:14.034 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.867814Z", 2021-09-02T07:42:14.035 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:14.035 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:14.035 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:14.036 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:14.036 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:42:14.036 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:14.036 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:14.037 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:14.037 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:14.037 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:14.037 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:14.037 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:14.038 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:14.038 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:14.038 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:14.038 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:14.039 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:42:14.039 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:14.039 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:14.039 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:14.040 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:14.040 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:14.040 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:14.040 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:14.040 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:14.041 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:14.041 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:14.041 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:42:14.041 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:14.042 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:42:14.042 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:42:14.042 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:42:14.042 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:14.042 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:42:14.044 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.867814_160.1 2021-09-02T07:42:19.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:19.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:19.046 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:42:19.107 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:42:19.108 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:42:19.175 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:42:19.175 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:42:19.176 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:42:19.176 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:42:19.176 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:42:19.176 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:42:19.177 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:19.177 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:19.177 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:42:19.177 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:19.178 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:42:19.178 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:19.178 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:19.178 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:42:19.178 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:19.179 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:42:19.179 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:19.179 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:42:19.179 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:19.180 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:19.180 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:19.180 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:19.180 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:42:19.180 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:19.181 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:19.181 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:42:19.181 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:19.181 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:19.182 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:19.182 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:42:19.182 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:19.182 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:19.183 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:19.183 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:19.183 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:42:19.183 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:19.183 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:19.184 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:19.188 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:19.188 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:19.188 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:19.188 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:19.188 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:19.189 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:19.189 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:19.189 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:19.189 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:42:19.190 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:19.190 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:19.190 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:19.190 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:19.191 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:19.191 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:19.191 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:19.191 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:19.191 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:19.192 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:19.192 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:42:19.192 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:19.192 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:42:19.193 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:42:19.193 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:42:19.193 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:19.193 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:42:19.195 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:42:24.196 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:24.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:24.198 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:42:24.285 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:42:24.286 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:42:24.380 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:42:24.381 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:42:24.381 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:42:24.381 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:42:24.382 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:42:24.382 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:42:24.382 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:24.382 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:24.382 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:42:24.383 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:24.383 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:42:24.383 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:24.383 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:24.384 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:42:24.384 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:24.384 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:42:24.384 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:24.385 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:42:24.385 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:24.385 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:24.385 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:24.385 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:24.386 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:42:24.386 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:24.386 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:24.386 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T07:42:24.387 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:24.387 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:24.387 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:24.387 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T07:42:24.387 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:24.388 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:24.388 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:24.388 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:24.388 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:42:24.389 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:24.389 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:24.389 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:24.389 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:24.390 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:24.390 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:24.390 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:24.390 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:24.390 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:24.391 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:24.391 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:24.391 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:42:24.391 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:24.392 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:24.392 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:24.392 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:24.392 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:24.392 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:24.393 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:24.393 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:24.393 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:24.393 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:24.394 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:42:24.394 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:24.394 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:42:24.394 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:42:24.395 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:42:24.395 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:24.395 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:42:24.396 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568127.103991_175.1 2021-09-02T07:42:29.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:29.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:29.399 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:42:29.446 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:42:29.446 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:42:29.509 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:42:29.510 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:42:29.510 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:42:29.510 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:42:29.510 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:42:29.511 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:42:29.511 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:29.511 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:29.511 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:42:29.512 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:29.512 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:42:29.512 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:29.512 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:29.513 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:42:29.513 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:29.513 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:42:29.513 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:29.514 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:42:29.514 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:29.514 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:29.514 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:29.514 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:29.515 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:42:29.515 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:29.515 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:29.515 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:42:29.516 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:29.516 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:29.516 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:29.516 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:42:29.517 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:29.517 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:29.517 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:29.517 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:29.518 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:42:29.518 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:29.518 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:29.518 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:29.519 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:29.519 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:29.519 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:29.519 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:29.519 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:29.520 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:29.520 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:29.520 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:29.520 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:42:29.521 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:29.521 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:29.521 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:29.521 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:29.522 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:29.522 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:29.522 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:29.522 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:29.523 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:29.523 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:29.523 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:42:29.523 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:29.523 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:42:29.524 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:42:29.524 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:42:29.524 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:29.524 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:42:29.526 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:42:34.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:34.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:34.531 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:42:34.575 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:42:34.576 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:42:34.649 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:42:34.649 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:42:34.650 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:42:34.650 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:42:34.650 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:42:34.650 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:42:34.651 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:34.651 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:34.651 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:42:34.651 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:34.652 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:42:34.652 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:34.652 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:34.652 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:42:34.653 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:34.653 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:42:34.653 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:34.654 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:42:34.654 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:34.654 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:34.654 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:34.655 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:34.655 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:42:34.655 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:34.655 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:34.656 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.844214_158.1", 2021-09-02T07:42:34.656 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:34.656 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:34.656 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:34.656 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.844214Z", 2021-09-02T07:42:34.657 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:34.657 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:34.657 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:34.658 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:34.658 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:42:34.658 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:34.658 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:34.659 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:34.659 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:34.659 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:34.659 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:34.660 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:34.660 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:34.660 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:34.660 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:34.660 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:34.661 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:42:34.661 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:34.661 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:34.661 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:34.662 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:34.662 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:34.662 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:34.662 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:34.662 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:34.663 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:34.663 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:34.663 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:42:34.663 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:34.664 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:42:34.664 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:42:34.664 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:42:34.664 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:34.665 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:42:34.666 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.844214_158.1 2021-09-02T07:42:39.668 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:39.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:39.670 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:42:39.716 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:42:39.716 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:42:39.784 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:42:39.784 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:42:39.784 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:42:39.784 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:42:39.785 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:42:39.785 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:42:39.785 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:39.785 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:39.786 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:42:39.786 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:39.786 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:42:39.786 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:39.787 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:39.787 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:42:39.787 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:39.787 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:42:39.788 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:39.788 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:42:39.788 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:39.788 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:39.789 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:39.789 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:39.789 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:42:39.789 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:39.789 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:39.790 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T07:42:39.790 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:39.790 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:39.790 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:39.791 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T07:42:39.791 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:39.792 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:39.792 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:39.792 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:39.792 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:42:39.793 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:39.793 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:39.793 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:39.793 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:39.794 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:39.794 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:39.794 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:39.794 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:39.794 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:39.795 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:39.795 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:39.795 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:42:39.795 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:39.796 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:39.796 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:39.796 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:39.796 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:39.796 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:39.797 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:39.797 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:39.797 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:39.797 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:39.798 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:42:39.798 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:39.798 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:42:39.798 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:42:39.798 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:42:39.799 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:39.799 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:42:39.800 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.875533_162.1 2021-09-02T07:42:44.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:44.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:44.803 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:42:44.848 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:42:44.848 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:42:44.915 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:42:44.915 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:42:44.916 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:42:44.916 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:42:44.916 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:42:44.916 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:42:44.916 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:44.917 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:44.917 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:42:44.917 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:44.917 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:42:44.918 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:44.918 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:44.918 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:42:44.918 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:44.919 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:42:44.919 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:44.919 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:42:44.919 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:44.920 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:44.920 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:44.920 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:44.920 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:42:44.920 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:44.921 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:44.921 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T07:42:44.922 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:44.922 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:44.922 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:44.923 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T07:42:44.923 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:44.923 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:44.924 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:44.924 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:44.924 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:42:44.924 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:44.924 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:44.925 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:44.925 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:44.925 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:44.925 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:44.926 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:44.926 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:44.926 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:44.927 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:44.927 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:44.927 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:42:44.927 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:44.927 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:44.928 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:44.928 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:44.928 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:44.928 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:44.929 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:44.929 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:44.929 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:44.929 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:44.929 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:42:44.930 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:44.930 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:42:44.930 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:42:44.930 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:42:44.931 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:44.931 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:42:44.933 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.875533_162.1 2021-09-02T07:42:49.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:49.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:49.936 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:42:49.981 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:42:49.982 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:42:50.061 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:42:50.062 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:42:50.062 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:42:50.062 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:42:50.062 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:42:50.063 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:42:50.063 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:50.063 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:50.063 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:42:50.064 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:50.064 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:42:50.064 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:50.064 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:50.064 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:42:50.065 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:50.065 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:42:50.065 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:50.065 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:42:50.066 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:50.066 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:50.066 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:50.066 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:50.067 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:42:50.067 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:50.067 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:50.067 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:42:50.067 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:50.068 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:50.068 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:50.068 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:42:50.068 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:50.069 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:50.069 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:50.069 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:50.069 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:42:50.069 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:50.070 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:50.070 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:50.070 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:50.070 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:50.071 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:50.071 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:50.071 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:50.071 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:50.072 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:50.072 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:50.072 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:42:50.072 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:50.073 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:50.073 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:50.073 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:50.073 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:50.073 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:50.074 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:50.074 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:50.074 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:50.075 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:50.075 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:42:50.075 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:50.075 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:42:50.075 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:42:50.076 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:42:50.076 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:50.076 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:42:50.077 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568130.037580_220.1 target=1_1630568126.837676_157.1 2021-09-02T07:42:55.080 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... FAIL 2021-09-02T07:42:55.081 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-39 2021-09-02T07:42:55.113 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=efybpk-40 2021-09-02T07:42:55.153 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=efybpk-41 2021-09-02T07:42:55.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:55.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:55.195 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:42:55.246 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:42:55.246 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:42:55.294 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T07:42:55.310 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:42:55.310 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:42:55.310 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:42:55.311 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:32:10.734248Z" 2021-09-02T07:42:55.311 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:42:55.311 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:42:55.311 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568575.180614_247.1", 2021-09-02T07:42:55.312 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:42:55.180614Z" 2021-09-02T07:42:55.312 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:42:55.312 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:42:55.312 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:42:55.313 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:42:55.313 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:42:55.313 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:42:55.313 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:42:55.313 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:42:55.314 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:42:55.314 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:42:55.315 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T07:42:55.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:55.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:42:55.317 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:42:55.360 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:42:55.360 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:42:55.440 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:42:55.441 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:42:55.441 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:42:55.441 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:42:55.441 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:42:55.442 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:42:55.442 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:55.442 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:55.442 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:42:55.443 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:55.443 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:42:55.443 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:55.443 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:55.443 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:42:55.444 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:55.444 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:42:55.444 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:55.444 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:42:55.445 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:55.446 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:55.446 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:55.446 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:55.446 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:42:55.447 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:55.447 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:55.447 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568575.100775_236.1", 2021-09-02T07:42:55.447 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:55.447 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:55.448 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:55.448 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:42:55.100775Z", 2021-09-02T07:42:55.448 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:55.449 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:55.449 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:55.450 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:55.450 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:42:55.450 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:55.451 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:55.451 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:55.451 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:55.451 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:55.452 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:55.452 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:55.452 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:55.452 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:55.453 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:55.453 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:42:55.453 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:42:55.453 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:42:55.453 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:42:55.454 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:42:55.454 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:42:55.454 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:42:55.454 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:42:55.455 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:42:55.455 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:42:55.455 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:55.455 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:55.456 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:42:55.456 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:42:55.456 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:42:55.456 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:42:55.457 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:42:55.457 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:42:55.457 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:42:55.459 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568575.100775_236.1 2021-09-02T07:43:00.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:00.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:00.461 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:43:00.510 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:43:00.510 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:43:00.583 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:43:00.583 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:43:00.583 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:43:00.583 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:43:00.584 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:43:00.584 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:43:00.584 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:00.584 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:00.585 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:43:00.585 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:00.585 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:43:00.585 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:00.586 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:00.586 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:43:00.586 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:00.586 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:43:00.587 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:00.587 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:43:00.587 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:00.587 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:00.587 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:00.588 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:00.588 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:43:00.588 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:00.588 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:00.589 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568575.180614_247.1", 2021-09-02T07:43:00.589 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:00.589 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:00.589 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:00.589 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:42:55.180614Z", 2021-09-02T07:43:00.590 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:00.590 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:00.590 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:00.590 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:00.591 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:43:00.591 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:00.591 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:00.591 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:00.591 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:00.592 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:00.592 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:00.592 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:00.592 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:00.592 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:00.593 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:00.593 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:00.593 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:43:00.593 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:00.594 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:00.594 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:00.594 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:00.594 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:00.594 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:00.595 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:00.595 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:00.595 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:00.595 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:00.596 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:43:00.596 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:00.596 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:43:00.596 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:43:00.596 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:43:00.597 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:00.597 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:43:00.599 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T07:43:00.599 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T07:43:00.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:00.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:00.601 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:43:00.651 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:43:00.651 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:43:00.729 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:43:00.729 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:43:00.730 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:43:00.730 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:43:00.730 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:43:00.730 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:43:00.731 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:43:00.731 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:43:00.731 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:43:00.731 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:43:00.732 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:43:00.732 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:43:00.732 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:43:00.732 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:43:00.732 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:43:00.733 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:43:00.733 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:43:00.733 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:43:00.733 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:43:00.734 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:43:00.734 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:43:00.734 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:43:00.734 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:43:00.734 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:43:00.735 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:43:00.735 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568575.180614_247.1", 2021-09-02T07:43:00.735 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:43:00.735 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:43:00.736 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:43:00.736 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:42:55.180614Z", 2021-09-02T07:43:00.736 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:43:00.736 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:43:00.736 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:43:00.737 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:43:00.737 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:43:00.737 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:43:00.737 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:43:00.738 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:43:00.738 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:43:00.738 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:43:00.738 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:43:00.738 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:43:00.739 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:43:00.739 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:43:00.739 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:43:00.739 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:43:00.740 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:43:00.740 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:43:00.740 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:43:00.740 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:43:00.740 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:43:00.741 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:43:00.741 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:43:00.741 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:43:00.741 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:43:00.742 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:43:00.742 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:43:00.742 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:43:00.742 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:43:00.743 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:43:00.743 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:43:00.743 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:43:00.743 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:43:00.744 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:43:00.745 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T07:43:00.745 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T07:43:00.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:00.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:00.747 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:43:00.792 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:43:00.793 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:43:00.858 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:43:00.859 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:43:00.859 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:43:00.859 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:43:00.859 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:43:00.860 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:43:00.860 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:00.860 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:00.860 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:43:00.861 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:00.861 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:43:00.861 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:00.861 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:00.861 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:43:00.862 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:00.862 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:43:00.862 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:00.862 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:43:00.863 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:00.863 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:00.863 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:00.863 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:00.863 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:43:00.864 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:00.864 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:00.864 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:43:00.864 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:00.865 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:00.865 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:00.865 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:43:00.865 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:00.866 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:00.866 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:00.866 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:00.866 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:43:00.866 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:00.867 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:00.867 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:00.867 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:00.867 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:00.868 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:00.868 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:00.868 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:00.869 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:00.869 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:00.869 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:00.869 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:43:00.870 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:00.870 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:00.870 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:00.870 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:00.871 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:00.871 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:00.871 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:00.871 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:00.871 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:00.872 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:00.872 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:43:00.872 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:00.872 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:43:00.873 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:43:00.873 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:43:00.873 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:00.873 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:43:00.875 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.837676_157.1 2021-09-02T07:43:05.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:05.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:05.878 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:43:05.930 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:43:05.931 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:43:05.996 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:43:05.997 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:43:05.997 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:43:05.998 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:43:05.998 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:43:05.999 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:43:05.999 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:06.000 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:06.000 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:43:06.001 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:06.001 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:43:06.001 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:06.002 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:06.002 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:43:06.002 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:06.002 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:43:06.003 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:06.003 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:43:06.003 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:06.003 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:06.004 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:06.004 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:06.005 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:43:06.005 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:06.005 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:06.005 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.835437_156.1", 2021-09-02T07:43:06.005 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:06.006 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:06.006 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:06.006 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.835437Z", 2021-09-02T07:43:06.006 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:06.007 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:06.007 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:06.007 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:06.007 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:43:06.008 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:06.008 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:06.008 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:06.008 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:06.009 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:06.009 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:06.009 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:06.009 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:06.009 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:06.010 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:06.010 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:06.010 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:43:06.011 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:06.011 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:06.011 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:06.011 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:06.012 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:06.012 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:06.012 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:06.012 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:06.012 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:06.014 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:06.014 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:43:06.014 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:06.015 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:43:06.015 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:43:06.015 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:43:06.016 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:06.016 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:43:06.019 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.835437_156.1 2021-09-02T07:43:11.020 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:11.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:11.022 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:43:11.068 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:43:11.069 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:43:11.134 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:43:11.134 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:43:11.134 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:43:11.135 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:43:11.135 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:43:11.135 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:43:11.136 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:11.136 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:11.137 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:43:11.137 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:11.137 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:43:11.138 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:11.138 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:11.138 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:43:11.138 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:11.139 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:43:11.139 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:11.139 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:43:11.140 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:11.140 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:11.140 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:11.140 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:11.141 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:43:11.141 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:11.141 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:11.141 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:43:11.142 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:11.143 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:11.144 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:11.144 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:43:11.144 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:11.145 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:11.145 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:11.145 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:11.146 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:43:11.146 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:11.146 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:11.146 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:11.147 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:11.147 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:11.147 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:11.147 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:11.147 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:11.148 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:11.148 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:11.148 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:11.148 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:43:11.149 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:11.149 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:11.149 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:11.149 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:11.150 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:11.150 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:11.150 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:11.150 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:11.151 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:11.151 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:11.151 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:43:11.151 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:11.151 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:43:11.152 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:43:11.152 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:43:11.152 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:11.153 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:43:11.155 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.837676_157.1 2021-09-02T07:43:16.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:16.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:16.158 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:43:16.200 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:43:16.200 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:43:16.271 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:43:16.271 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:43:16.272 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:43:16.272 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:43:16.272 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:43:16.272 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:43:16.273 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:16.273 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:16.273 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:43:16.274 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:16.274 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:43:16.274 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:16.274 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:16.275 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:43:16.276 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:16.276 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:43:16.276 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:16.277 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:43:16.278 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:16.279 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:16.279 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:16.279 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:16.279 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:43:16.280 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:16.280 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:16.280 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.911309_165.1", 2021-09-02T07:43:16.280 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:16.281 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:16.281 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:16.281 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.911309Z", 2021-09-02T07:43:16.281 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:16.282 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:16.282 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:16.282 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:16.282 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:43:16.282 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:16.283 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:16.283 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:16.283 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:16.284 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:16.284 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:16.284 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:16.285 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:16.285 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:16.285 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:16.285 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:16.286 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:43:16.286 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:16.286 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:16.286 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:16.287 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:16.288 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:16.288 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:16.288 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:16.289 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:16.289 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:16.289 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:16.289 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:43:16.289 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:16.290 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:43:16.290 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:43:16.290 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:43:16.290 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:16.291 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:43:16.292 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.911309_165.1 2021-09-02T07:43:21.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:21.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:21.295 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:43:21.351 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:43:21.351 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:43:21.770 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:43:21.770 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:43:21.771 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:43:21.771 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:43:21.773 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:43:21.773 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:43:21.773 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:21.774 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:21.774 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:43:21.774 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:21.774 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:43:21.775 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:21.775 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:21.775 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:43:21.776 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:21.776 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:43:21.776 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:21.776 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:43:21.777 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:21.777 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:21.777 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:21.777 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:21.778 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:43:21.778 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:21.778 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:21.778 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568129.876241_207.1", 2021-09-02T07:43:21.779 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:21.779 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:21.779 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:21.779 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:29.876241Z", 2021-09-02T07:43:21.779 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:21.780 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:21.780 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:21.784 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:21.784 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:43:21.784 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:21.784 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:21.785 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:21.785 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:21.785 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:21.785 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:21.786 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:21.786 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:21.786 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:21.786 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:21.787 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:21.787 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:43:21.787 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:21.787 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:21.787 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:21.788 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:21.788 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:21.788 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:21.788 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:21.789 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:21.790 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:21.790 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:21.791 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:43:21.791 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:21.791 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:43:21.791 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:43:21.791 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:43:21.792 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:21.792 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:43:21.793 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568129.876241_207.1 2021-09-02T07:43:26.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:26.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:26.797 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:43:26.851 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:43:26.851 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:43:27.073 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:43:27.073 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:43:27.074 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:43:27.074 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:43:27.074 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:43:27.075 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:43:27.075 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:27.075 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:27.075 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:43:27.076 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:27.076 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:43:27.076 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:27.076 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:27.077 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:43:27.077 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:27.077 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:43:27.078 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:27.078 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:43:27.079 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:27.079 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:27.079 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:27.079 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:27.080 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:43:27.080 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:27.080 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:27.080 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.862512_159.1", 2021-09-02T07:43:27.080 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:27.081 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:27.081 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:27.081 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.862512Z", 2021-09-02T07:43:27.081 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:27.082 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:27.082 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:27.082 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:27.082 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:43:27.083 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:27.083 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:27.083 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:27.083 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:27.083 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:27.084 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:27.084 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:27.084 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:27.084 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:27.085 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:27.085 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:27.085 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:43:27.085 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:27.085 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:27.086 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:27.086 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:27.086 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:27.086 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:27.087 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:27.087 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:27.087 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:27.087 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:27.088 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:43:27.088 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:27.089 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:43:27.089 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:43:27.089 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:43:27.089 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:27.089 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:43:27.091 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.862512_159.1 2021-09-02T07:43:32.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:32.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:32.094 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:43:32.136 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:43:32.137 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:43:32.208 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:43:32.208 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:43:32.208 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:43:32.209 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:43:32.209 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:43:32.209 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:43:32.209 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:32.210 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:32.210 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:43:32.210 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:32.210 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:43:32.211 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:32.211 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:32.211 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:43:32.211 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:32.212 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:43:32.212 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:32.212 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:43:32.213 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:32.213 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:32.213 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:32.213 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:32.214 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:43:32.214 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:32.214 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:32.214 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.869904_161.1", 2021-09-02T07:43:32.215 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:32.215 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:32.215 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:32.215 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.869904Z", 2021-09-02T07:43:32.215 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:32.216 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:32.216 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:32.216 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:32.216 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:43:32.217 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:32.217 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:32.218 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:32.218 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:32.218 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:32.218 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:32.219 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:32.219 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:32.219 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:32.219 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:32.220 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:32.220 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:43:32.220 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:32.220 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:32.221 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:32.221 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:32.221 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:32.221 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:32.221 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:32.222 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:32.222 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:32.222 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:32.223 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:43:32.223 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:32.223 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:43:32.223 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:43:32.224 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:43:32.224 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:32.224 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:43:32.225 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.869904_161.1 2021-09-02T07:43:37.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:37.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:37.230 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:43:37.349 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:43:37.350 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:43:37.637 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:43:37.637 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:43:37.637 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:43:37.638 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:43:37.638 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:43:37.638 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:43:37.638 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:37.638 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:37.639 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:43:37.639 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:37.639 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:43:37.639 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:37.640 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:37.640 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:43:37.640 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:37.640 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:43:37.641 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:37.641 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:43:37.641 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:37.641 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:37.641 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:37.642 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:37.642 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:43:37.642 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:37.642 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:37.643 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:43:37.643 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:37.643 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:37.644 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:37.644 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:43:37.645 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:37.645 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:37.645 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:37.645 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:37.645 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:43:37.646 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:37.646 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:37.646 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:37.647 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:37.647 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:37.647 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:37.647 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:37.648 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:37.648 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:37.649 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:37.649 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:37.649 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:43:37.650 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:37.650 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:37.650 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:37.650 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:37.650 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:37.651 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:37.651 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:37.651 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:37.651 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:37.652 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:37.652 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:43:37.652 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:37.652 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:43:37.653 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:43:37.653 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:43:37.653 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:37.653 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:43:37.655 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.837676_157.1 2021-09-02T07:43:42.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:42.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:42.657 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:43:42.750 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:43:42.751 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:43:43.180 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:43:43.181 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:43:43.181 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:43:43.181 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:43:43.182 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:43:43.182 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:43:43.182 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:43.182 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:43.183 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:43:43.183 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:43.183 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:43:43.183 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:43.184 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:43.184 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:43:43.184 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:43.184 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:43:43.184 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:43.185 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:43:43.185 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:43.185 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:43.185 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:43.186 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:43.186 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:43:43.186 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:43.186 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:43.187 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.869904_161.1", 2021-09-02T07:43:43.187 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:43.187 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:43.187 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:43.187 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.869904Z", 2021-09-02T07:43:43.188 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:43.188 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:43.188 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:43.188 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:43.189 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:43:43.189 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:43.189 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:43.190 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:43.190 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:43.190 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:43.190 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:43.190 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:43.191 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:43.191 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:43.191 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:43.191 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:43.192 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:43:43.192 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:43.192 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:43.192 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:43.192 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:43.193 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:43.193 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:43.193 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:43.193 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:43.194 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:43.194 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:43.194 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:43:43.194 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:43.194 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:43:43.195 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:43:43.195 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:43:43.195 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:43.195 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:43:43.196 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.869904_161.1 2021-09-02T07:43:48.198 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:48.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:48.199 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:43:48.286 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:43:48.287 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:43:48.540 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:43:48.541 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:43:48.541 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:43:48.542 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:43:48.542 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:43:48.542 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:43:48.542 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:48.543 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:48.543 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:43:48.543 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:48.543 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:43:48.544 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:48.544 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:48.544 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:43:48.544 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:48.545 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:43:48.545 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:48.545 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:43:48.545 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:48.545 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:48.546 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:48.546 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:48.546 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:43:48.546 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:48.547 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:48.547 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.102736_174.1", 2021-09-02T07:43:48.547 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:48.547 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:48.547 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:48.548 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.102736Z", 2021-09-02T07:43:48.548 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:48.549 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:48.549 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:48.549 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:48.549 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:43:48.550 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:48.550 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:48.550 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:48.550 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:48.551 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:48.551 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:48.551 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:48.551 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:48.552 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:48.552 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:48.553 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:48.553 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:43:48.553 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:48.553 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:48.553 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:48.554 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:48.554 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:48.555 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:48.555 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:48.555 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:48.555 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:48.555 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:48.556 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:43:48.556 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:48.556 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:43:48.557 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:43:48.557 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:43:48.557 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:48.557 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:43:48.558 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568127.102736_174.1 2021-09-02T07:43:53.560 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:53.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:53.562 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:43:53.649 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:43:53.650 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:43:54.081 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:43:54.082 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:43:54.082 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:43:54.082 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:43:54.083 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:43:54.083 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:43:54.083 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:54.083 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:54.084 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:43:54.084 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:54.084 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:43:54.084 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:54.085 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:54.085 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:43:54.085 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:54.085 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:43:54.085 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:54.086 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:43:54.086 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:54.086 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:54.086 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:54.087 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:54.087 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:43:54.087 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:54.087 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:54.088 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:43:54.088 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:54.088 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:54.089 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:54.089 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:43:54.089 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:54.089 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:54.090 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:54.090 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:54.090 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:43:54.090 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:54.091 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:54.091 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:54.092 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:54.092 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:54.092 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:54.092 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:54.093 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:54.093 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:54.093 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:54.093 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:54.094 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:43:54.094 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:54.094 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:54.094 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:54.095 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:54.095 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:54.095 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:54.095 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:54.096 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:54.096 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:54.096 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:54.096 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:43:54.097 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:54.097 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:43:54.097 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:43:54.097 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:43:54.098 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:54.098 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:43:54.099 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.837676_157.1 2021-09-02T07:43:59.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:59.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:43:59.102 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:43:59.182 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:43:59.183 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:43:59.439 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:43:59.439 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:43:59.440 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:43:59.440 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:43:59.440 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:43:59.440 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:43:59.441 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:59.441 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:59.441 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:43:59.442 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:59.442 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:43:59.442 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:59.442 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:59.442 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:43:59.443 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:59.443 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:43:59.443 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:59.443 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:43:59.444 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:59.444 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:59.444 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:59.444 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:59.444 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:43:59.445 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:59.445 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:59.445 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:43:59.445 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:59.446 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:59.446 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:59.446 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:43:59.446 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:59.446 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:59.447 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:59.447 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:59.447 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:43:59.447 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:59.448 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:59.449 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:59.449 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:59.449 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:59.449 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:59.450 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:59.450 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:59.450 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:59.451 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:59.451 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:43:59.451 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:43:59.451 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:43:59.452 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:43:59.452 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:43:59.452 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:43:59.452 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:43:59.453 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:43:59.453 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:43:59.453 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:43:59.453 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:59.453 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:59.454 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:43:59.454 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:43:59.454 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:43:59.455 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:43:59.455 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:43:59.455 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:43:59.455 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:43:59.458 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.837676_157.1 2021-09-02T07:44:00.011 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:44:00.012+0000 7f968f999700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-02T07:44:00.026 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:44:00.015+0000 7f968f999700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-02T07:44:04.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:04.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:04.461 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:44:04.571 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:44:04.572 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:44:04.955 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:44:04.956 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:44:04.957 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:44:04.957 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:44:04.957 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:44:04.957 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:44:04.958 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:04.958 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:04.958 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:44:04.958 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:04.959 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:44:04.959 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:04.959 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:04.959 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:44:04.959 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:04.960 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:44:04.960 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:04.960 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:44:04.960 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:04.961 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:04.961 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:04.961 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:04.961 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:44:04.961 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:04.962 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:04.962 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:44:04.962 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:04.962 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:04.963 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:04.963 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:44:04.963 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:04.963 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:04.963 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:04.964 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:04.964 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:44:04.964 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:04.964 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:04.965 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:04.965 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:04.965 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:04.965 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:04.965 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:04.966 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:04.966 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:04.967 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:04.967 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:04.968 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:44:04.968 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:04.968 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:04.968 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:04.968 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:04.969 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:04.969 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:04.969 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:04.969 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:04.970 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:04.970 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:04.970 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:44:04.970 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:04.971 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:44:04.971 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:44:04.971 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:44:04.971 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:04.972 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:44:04.973 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.837676_157.1 2021-09-02T07:44:09.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:09.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:09.975 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:44:10.076 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:44:10.077 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:44:10.426 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:44:10.426 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:44:10.427 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:44:10.427 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:44:10.427 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:44:10.427 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:44:10.427 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:10.428 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:10.428 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:44:10.428 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:10.428 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:44:10.429 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:10.429 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:10.429 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:44:10.429 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:10.429 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:44:10.430 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:10.430 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:44:10.430 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:10.430 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:10.431 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:10.431 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:10.431 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:44:10.431 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:10.431 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:10.432 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.835437_156.1", 2021-09-02T07:44:10.432 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:10.432 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:10.432 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:10.433 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.835437Z", 2021-09-02T07:44:10.433 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:10.433 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:10.433 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:10.433 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:10.434 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:44:10.434 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:10.434 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:10.434 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:10.434 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:10.435 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:10.435 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:10.435 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:10.435 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:10.436 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:10.436 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:10.436 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:10.436 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:44:10.436 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:10.437 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:10.437 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:10.437 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:10.437 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:10.438 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:10.438 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:10.438 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:10.438 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:10.438 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:10.439 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:44:10.439 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:10.439 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:44:10.439 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:44:10.440 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:44:10.440 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:10.440 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:44:10.441 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.835437_156.1 2021-09-02T07:44:15.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:15.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:15.444 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:44:15.536 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:44:15.536 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:44:16.009 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:44:16.010 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:44:16.010 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:44:16.010 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:44:16.011 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:44:16.011 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:44:16.011 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:16.011 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:16.012 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:44:16.012 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:16.012 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:44:16.012 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:16.013 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:16.013 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:44:16.013 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:16.013 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:44:16.014 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:16.014 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:44:16.014 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:16.014 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:16.014 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:16.015 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:16.015 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:44:16.015 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:16.015 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:16.016 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T07:44:16.016 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:16.016 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:16.016 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:16.017 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T07:44:16.017 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:16.017 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:16.017 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:16.017 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:16.018 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:44:16.018 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:16.018 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:16.018 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:16.019 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:16.019 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:16.019 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:16.019 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:16.019 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:16.020 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:16.020 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:16.020 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:16.020 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:44:16.021 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:16.021 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:16.021 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:16.021 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:16.021 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:16.022 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:16.022 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:16.022 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:16.023 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:16.023 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:16.023 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:44:16.023 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:16.024 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:44:16.024 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:44:16.024 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:44:16.024 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:16.024 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:44:16.026 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.906521_163.1 2021-09-02T07:44:20.014 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:44:20.015+0000 7f968f999700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-02T07:44:21.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:21.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:21.029 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:44:21.124 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:44:21.125 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:44:21.483 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:44:21.483 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:44:21.484 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:44:21.484 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:44:21.485 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:44:21.485 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:44:21.485 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:21.485 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:21.486 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:44:21.486 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:21.486 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:44:21.486 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:21.487 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:21.487 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:44:21.487 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:21.487 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:44:21.487 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:21.488 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:44:21.488 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:21.488 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:21.488 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:21.489 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:21.489 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:44:21.489 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:21.489 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:21.489 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:44:21.490 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:21.490 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:21.490 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:21.490 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:44:21.491 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:21.491 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:21.491 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:21.491 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:21.491 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:44:21.492 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:21.492 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:21.492 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:21.492 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:21.493 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:21.493 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:21.493 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:21.494 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:21.494 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:21.495 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:21.495 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:21.495 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:44:21.495 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:21.496 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:21.496 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:21.496 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:21.496 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:21.497 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:21.497 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:21.497 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:21.497 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:21.497 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:21.498 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:44:21.498 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:21.498 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:44:21.498 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:44:21.499 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:44:21.499 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:21.499 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:44:21.500 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.837676_157.1 2021-09-02T07:44:26.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:26.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:26.503 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:44:26.584 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:44:26.584 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:44:26.934 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:44:26.934 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:44:26.934 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:44:26.935 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:44:26.935 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:44:26.935 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:44:26.935 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:26.936 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:26.936 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:44:26.936 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:26.936 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:44:26.936 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:26.937 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:26.937 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:44:26.937 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:26.937 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:44:26.938 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:26.938 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:44:26.938 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:26.938 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:26.939 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:26.939 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:26.939 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:44:26.939 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:26.939 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:26.940 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:44:26.940 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:26.940 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:26.940 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:26.941 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:44:26.941 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:26.941 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:26.941 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:26.941 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:26.942 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:44:26.942 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:26.942 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:26.942 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:26.943 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:26.943 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:26.943 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:26.943 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:26.943 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:26.944 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:26.944 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:26.944 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:26.945 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:44:26.945 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:26.945 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:26.945 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:26.946 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:26.946 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:26.946 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:26.946 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:26.946 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:26.947 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:26.947 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:26.947 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:44:26.947 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:26.948 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:44:26.948 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:44:26.948 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:44:26.949 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:26.949 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:44:26.950 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.837676_157.1 2021-09-02T07:44:31.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:31.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:31.953 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:44:32.028 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:44:32.028 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:44:32.404 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:44:32.405 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:44:32.405 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:44:32.405 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:44:32.406 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:44:32.406 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:44:32.406 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:32.406 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:32.407 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:44:32.407 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:32.407 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:44:32.407 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:32.408 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:32.408 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:44:32.408 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:32.408 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:44:32.409 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:32.409 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:44:32.409 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:32.409 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:32.409 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:32.410 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:32.410 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:44:32.410 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:32.410 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:32.411 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.835437_156.1", 2021-09-02T07:44:32.411 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:32.411 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:32.411 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:32.411 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.835437Z", 2021-09-02T07:44:32.412 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:32.412 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:32.412 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:32.412 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:32.413 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:44:32.413 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:32.415 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:32.415 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:32.416 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:32.416 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:32.416 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:32.416 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:32.417 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:32.417 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:32.417 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:32.417 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:32.417 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:44:32.418 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:32.418 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:32.418 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:32.418 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:32.419 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:32.419 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:32.419 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:32.419 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:32.420 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:32.420 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:32.420 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:44:32.420 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:32.421 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:44:32.421 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:44:32.421 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:44:32.421 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:32.421 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:44:32.423 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.835437_156.1 2021-09-02T07:44:37.425 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:37.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:37.426 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:44:37.504 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:44:37.504 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:44:38.033 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:44:38.034 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:44:38.034 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:44:38.034 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:44:38.035 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:44:38.035 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:44:38.035 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:38.035 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:38.036 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:44:38.036 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:38.036 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:44:38.036 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:38.037 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:38.037 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:44:38.038 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:38.038 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:44:38.038 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:38.039 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:44:38.039 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:38.039 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:38.039 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:38.040 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:38.040 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:44:38.040 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:38.041 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:38.041 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.911309_165.1", 2021-09-02T07:44:38.041 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:38.041 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:38.042 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:38.042 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.911309Z", 2021-09-02T07:44:38.042 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:38.042 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:38.043 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:38.043 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:38.043 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:44:38.043 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:38.044 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:38.044 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:38.044 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:38.044 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:38.045 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:38.045 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:38.045 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:38.045 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:38.046 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:38.046 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:38.046 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:44:38.046 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:38.047 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:38.047 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:38.047 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:38.048 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:38.048 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:38.048 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:38.048 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:38.049 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:38.049 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:38.049 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:44:38.049 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:38.050 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:44:38.050 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:44:38.050 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:44:38.050 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:38.051 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:44:38.052 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.911309_165.1 2021-09-02T07:44:43.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:43.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:43.055 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:44:43.180 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:44:43.180 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:44:43.713 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:44:43.713 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:44:43.714 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:44:43.714 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:44:43.714 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:44:43.715 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:44:43.716 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:43.716 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:43.716 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:44:43.717 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:43.717 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:44:43.717 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:43.718 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:43.718 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:44:43.720 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:43.720 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:44:43.720 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:43.721 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:44:43.721 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:43.722 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:43.722 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:43.722 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:43.722 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:44:43.723 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:43.723 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:43.724 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.869904_161.1", 2021-09-02T07:44:43.724 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:43.724 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:43.724 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:43.725 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.869904Z", 2021-09-02T07:44:43.725 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:43.725 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:43.725 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:43.726 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:43.726 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:44:43.727 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:43.727 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:43.727 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:43.728 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:43.728 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:43.728 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:43.729 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:43.729 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:43.729 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:43.729 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:43.730 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:43.730 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:44:43.731 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:43.731 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:43.731 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:43.732 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:43.732 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:43.732 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:43.732 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:43.733 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:43.733 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:43.733 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:43.734 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:44:43.734 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:43.734 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:44:43.734 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:44:43.735 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:44:43.735 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:43.735 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:44:43.737 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.869904_161.1 2021-09-02T07:44:48.738 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:48.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:48.739 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:44:48.846 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:44:48.846 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:44:49.205 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:44:49.205 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:44:49.206 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:44:49.206 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:44:49.206 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:44:49.206 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:44:49.207 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:49.207 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:49.207 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:44:49.207 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:49.208 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:44:49.208 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:49.208 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:49.208 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:44:49.208 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:49.209 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:44:49.209 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:49.209 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:44:49.209 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:49.210 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:49.210 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:49.210 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:49.210 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:44:49.211 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:49.211 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:49.211 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:44:49.211 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:49.212 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:49.212 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:49.212 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:44:49.212 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:49.212 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:49.213 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:49.213 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:49.213 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:44:49.213 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:49.214 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:49.214 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:49.214 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:49.214 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:49.214 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:49.215 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:49.215 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:49.215 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:49.215 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:49.216 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:49.216 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:44:49.216 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:49.216 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:49.216 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:49.217 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:49.217 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:49.217 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:49.217 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:49.218 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:49.218 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:49.218 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:49.218 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:44:49.218 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:49.219 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:44:49.219 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:44:49.219 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:44:49.219 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:49.220 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:44:49.221 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.837676_157.1 2021-09-02T07:44:54.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:54.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:54.224 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:44:54.304 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:44:54.305 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:44:54.558 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:44:54.558 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:44:54.558 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:44:54.559 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:44:54.559 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:44:54.560 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:44:54.560 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:54.560 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:54.560 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:44:54.561 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:54.561 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:44:54.561 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:54.562 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:54.562 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:44:54.563 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:54.563 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:44:54.563 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:54.564 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:44:54.564 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:54.564 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:54.564 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:54.565 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:54.565 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:44:54.565 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:54.565 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:54.566 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:44:54.566 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:54.566 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:54.566 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:54.567 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:44:54.567 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:54.567 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:54.567 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:54.568 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:54.568 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:44:54.568 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:54.568 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:54.568 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:54.569 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:54.570 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:54.570 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:54.570 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:54.571 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:54.571 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:54.571 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:54.572 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:54.573 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:44:54.573 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:54.573 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:54.574 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:54.591 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:54.592 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:54.592 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:54.594 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:54.595 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:54.595 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:54.595 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:54.595 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:44:54.596 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:54.596 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:44:54.596 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:44:54.596 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:44:54.596 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:54.597 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:44:54.598 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.837676_157.1 2021-09-02T07:44:59.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:59.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:44:59.601 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:44:59.687 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:44:59.687 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:44:59.904 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:44:59.904 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:44:59.904 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:44:59.905 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:44:59.905 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:44:59.905 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:44:59.905 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:59.905 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:59.907 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:44:59.907 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:59.908 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:44:59.908 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:59.908 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:59.908 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:44:59.908 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:59.909 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:44:59.909 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:59.909 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:44:59.909 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:59.910 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:59.910 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:59.910 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:59.910 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:44:59.911 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:59.911 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:59.911 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:44:59.911 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:59.911 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:59.912 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:59.912 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:44:59.912 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:59.912 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:59.913 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:59.913 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:59.913 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:44:59.913 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:59.913 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:59.914 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:59.914 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:59.914 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:59.914 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:59.915 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:59.915 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:59.915 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:59.915 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:59.915 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:44:59.916 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:44:59.916 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:44:59.917 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:44:59.917 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:44:59.918 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:44:59.918 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:44:59.918 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:44:59.918 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:44:59.919 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:44:59.919 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:59.919 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:59.919 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:44:59.920 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:44:59.920 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:44:59.920 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:44:59.920 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:44:59.920 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:44:59.921 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:44:59.922 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.837676_157.1 2021-09-02T07:45:04.924 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:45:04.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:45:04.926 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:45:05.017 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:45:05.018 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:45:05.375 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:45:05.385 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:45:05.385 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:45:05.386 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:45:05.386 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:45:05.387 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:45:05.387 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:05.387 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:05.388 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:45:05.388 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:05.389 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:45:05.389 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:05.389 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:05.390 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:45:05.390 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:05.399 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:45:05.400 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:05.400 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:45:05.400 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:05.401 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:05.401 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:05.402 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:05.403 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:45:05.403 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:05.404 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:05.404 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.869904_161.1", 2021-09-02T07:45:05.405 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:05.405 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:05.405 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:05.406 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.869904Z", 2021-09-02T07:45:05.406 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:05.407 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:05.408 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:05.408 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:05.409 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:45:05.409 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:05.409 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:05.410 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:45:05.410 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:05.410 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:05.410 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:05.411 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:45:05.411 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:05.411 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:05.411 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:05.411 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:05.412 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:45:05.412 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:05.412 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:05.412 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:45:05.413 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:05.413 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:05.413 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:05.413 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:45:05.414 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:05.414 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:05.414 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:05.414 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:45:05.415 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:05.415 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:45:05.415 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:45:05.415 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:45:05.415 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:05.416 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:45:05.417 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.869904_161.1 2021-09-02T07:45:10.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:45:10.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:45:10.423 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:45:10.521 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:45:10.522 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:45:10.914 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:45:10.915 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:45:10.915 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:45:10.915 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:45:10.916 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:45:10.916 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:45:10.916 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:10.917 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:10.917 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:45:10.918 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:10.918 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:45:10.918 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:10.919 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:10.919 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:45:10.919 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:10.920 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:45:10.920 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:10.921 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:45:10.921 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:10.921 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:10.922 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:10.922 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:10.923 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:45:10.923 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:10.923 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:10.924 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:45:10.924 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:10.925 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:10.925 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:10.925 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:45:10.926 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:10.926 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:10.927 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:10.927 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:10.928 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:45:10.928 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:10.929 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:10.929 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:45:10.930 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:10.930 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:10.930 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:10.931 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:45:10.931 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:10.931 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:10.932 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:10.933 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:10.934 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:45:10.934 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:10.935 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:10.935 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:45:10.936 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:10.936 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:10.937 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:10.937 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:45:10.937 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:10.938 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:10.938 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:10.938 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:45:10.938 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:10.939 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:45:10.939 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:45:10.939 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:45:10.939 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:10.939 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:45:10.941 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.837676_157.1 2021-09-02T07:45:15.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:45:15.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:45:15.945 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:45:16.010 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:45:16.010 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:45:16.657 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:45:16.658 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:45:16.658 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:45:16.658 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:45:16.659 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:45:16.659 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:45:16.659 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:16.659 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:16.660 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:45:16.660 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:16.660 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:45:16.660 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:16.661 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:16.661 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:45:16.661 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:16.661 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:45:16.662 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:16.662 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:45:16.662 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:16.662 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:16.663 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:16.663 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:16.663 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:45:16.663 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:16.664 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:16.664 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.869904_161.1", 2021-09-02T07:45:16.664 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:16.664 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:16.665 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:16.665 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.869904Z", 2021-09-02T07:45:16.665 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:16.665 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:16.665 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:16.666 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:16.666 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:45:16.666 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:16.667 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:16.667 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:45:16.667 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:16.667 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:16.668 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:16.668 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:45:16.668 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:16.668 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:16.668 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:16.669 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:16.669 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:45:16.669 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:16.669 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:16.670 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:45:16.670 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:16.670 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:16.670 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:16.671 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:45:16.671 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:16.671 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:16.671 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:16.672 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:45:16.672 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:16.672 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:45:16.672 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:45:16.672 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:45:16.673 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:16.673 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:45:16.674 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.869904_161.1 2021-09-02T07:45:21.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:45:21.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:45:21.678 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:45:21.766 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:45:21.766 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:45:22.317 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:45:22.317 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:45:22.317 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:45:22.318 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:45:22.318 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:45:22.318 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:45:22.319 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:22.319 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:22.319 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:45:22.319 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:22.320 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:45:22.320 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:22.320 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:22.320 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:45:22.320 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:22.321 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:45:22.321 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:22.321 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:45:22.321 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:22.322 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:22.322 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:22.322 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:22.322 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:45:22.323 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:22.323 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:22.323 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:45:22.323 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:22.323 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:22.324 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:22.324 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:45:22.324 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:22.324 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:22.325 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:22.325 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:22.325 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:45:22.325 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:22.325 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:22.326 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:45:22.326 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:22.326 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:22.327 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:22.327 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:45:22.327 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:22.327 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:22.328 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:22.328 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:22.328 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:45:22.328 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:22.329 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:22.329 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:45:22.329 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:22.330 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:22.330 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:22.330 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:45:22.330 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:22.330 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:22.331 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:22.331 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:45:22.331 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:22.332 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:45:22.332 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:45:22.332 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:45:22.332 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:22.333 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:45:22.333 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.837676_157.1 2021-09-02T07:45:27.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:45:27.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:45:27.337 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:45:27.410 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:45:27.410 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:45:28.044 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:45:28.045 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:45:28.045 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:45:28.045 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:45:28.046 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:45:28.046 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:45:28.046 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:28.046 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:28.046 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:45:28.047 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:28.047 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:45:28.047 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:28.047 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:28.048 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:45:28.048 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:28.048 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:45:28.048 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:28.049 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:45:28.049 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:28.049 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:28.049 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:28.050 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:28.050 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:45:28.050 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:28.050 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:28.050 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.844214_158.1", 2021-09-02T07:45:28.051 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:28.051 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:28.051 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:28.051 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.844214Z", 2021-09-02T07:45:28.052 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:28.052 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:28.052 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:28.052 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:28.052 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:45:28.053 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:28.053 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:28.053 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:45:28.053 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:28.054 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:28.054 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:28.054 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:45:28.054 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:28.055 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:28.055 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:28.055 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:28.055 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:45:28.055 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:28.056 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:28.056 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:45:28.056 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:28.056 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:28.057 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:28.057 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:45:28.057 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:28.057 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:28.057 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:28.058 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:45:28.058 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:28.058 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:45:28.058 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:45:28.059 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:45:28.059 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:28.059 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:45:28.060 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.844214_158.1 2021-09-02T07:45:33.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:45:33.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:45:33.069 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:45:33.167 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:45:33.168 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:45:33.648 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:45:33.649 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:45:33.650 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:45:33.651 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:45:33.651 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:45:33.651 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:45:33.652 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:33.652 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:33.653 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:45:33.653 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:33.653 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:45:33.654 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:33.654 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:33.655 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:45:33.655 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:33.655 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:45:33.656 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:33.657 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:45:33.657 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:33.658 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:33.658 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:33.658 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:33.659 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:45:33.659 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:33.660 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:33.660 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.835437_156.1", 2021-09-02T07:45:33.660 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:33.667 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:33.668 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:33.668 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.835437Z", 2021-09-02T07:45:33.668 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:33.669 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:33.669 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:33.669 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:33.670 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:45:33.670 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:33.670 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:33.670 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:45:33.670 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:33.671 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:33.671 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:33.671 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:45:33.672 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:33.672 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:33.672 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:33.672 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:33.673 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:45:33.673 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:33.673 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:33.673 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:45:33.673 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:33.674 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:33.674 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:33.674 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:45:33.674 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:33.675 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:33.675 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:33.675 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:45:33.675 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:33.675 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:45:33.676 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:45:33.676 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:45:33.676 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:33.676 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:45:33.701 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.835437_156.1 2021-09-02T07:45:38.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:45:38.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:45:38.722 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:45:39.012 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:45:39.012 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:45:39.117 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:45:39.117 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:45:39.117 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:45:39.118 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:45:39.118 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:45:39.119 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:45:39.119 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:39.119 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:39.119 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:45:39.119 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:39.120 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:45:39.120 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:39.120 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:39.120 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:45:39.121 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:39.121 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:45:39.121 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:39.121 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:45:39.121 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:39.122 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:39.122 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:39.122 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:39.122 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:45:39.123 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:39.123 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:39.123 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.867814_160.1", 2021-09-02T07:45:39.123 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:39.124 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:39.124 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:39.124 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.867814Z", 2021-09-02T07:45:39.124 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:39.124 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:39.125 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:39.125 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:39.125 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:45:39.125 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:39.126 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:39.126 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:45:39.126 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:39.127 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:39.127 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:39.127 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:45:39.128 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:39.131 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:39.131 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:39.132 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:39.132 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:45:39.133 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:39.133 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:39.133 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:45:39.134 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:39.134 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:39.135 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:39.135 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:45:39.136 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:39.138 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:39.138 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:39.139 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:45:39.139 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:39.140 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:45:39.140 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:45:39.141 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:45:39.141 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:39.141 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:45:39.142 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.867814_160.1 2021-09-02T07:45:44.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:45:44.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:45:44.149 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:45:44.254 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:45:44.255 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:45:44.406 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:45:44.406 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:45:44.407 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:45:44.407 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:45:44.407 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:45:44.407 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:45:44.408 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:44.408 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:44.408 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:45:44.408 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:44.409 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:45:44.409 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:44.409 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:44.409 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:45:44.410 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:44.410 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:45:44.410 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:44.410 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:45:44.410 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:44.411 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:44.411 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:44.411 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:44.411 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:45:44.412 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:44.412 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:44.412 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.837676_157.1", 2021-09-02T07:45:44.412 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:44.412 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:44.413 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:44.413 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.837676Z", 2021-09-02T07:45:44.413 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:44.413 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:44.414 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:44.414 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:44.414 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:45:44.414 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:44.414 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:44.415 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:45:44.415 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:44.415 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:44.415 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:44.416 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:45:44.416 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:44.416 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:44.416 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:44.417 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:44.417 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:45:44.417 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:44.417 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:44.417 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:45:44.418 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:44.418 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:44.418 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:44.418 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:45:44.419 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:44.419 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:44.419 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:44.419 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:45:44.420 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:44.420 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:45:44.420 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:45:44.420 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:45:44.420 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:44.421 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:45:44.422 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.837676_157.1 2021-09-02T07:45:49.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:45:49.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:45:49.425 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:45:49.504 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:45:49.504 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:45:49.763 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:45:49.763 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:45:49.763 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:45:49.764 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:45:49.764 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:45:49.764 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:45:49.764 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:49.765 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:49.765 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:45:49.765 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:49.765 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:45:49.765 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:49.766 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:49.766 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:45:49.766 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:49.766 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:45:49.767 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:49.767 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:45:49.767 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:49.770 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:49.773 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:49.773 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:49.773 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:45:49.774 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:49.775 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:49.779 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T07:45:49.779 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:49.780 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:49.783 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:49.783 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T07:45:49.783 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:49.783 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:49.784 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:49.784 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:49.784 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:45:49.784 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:49.784 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:49.785 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:45:49.785 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:49.785 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:49.785 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:49.786 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:45:49.786 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:49.786 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:49.786 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:49.786 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:49.787 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:45:49.787 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:49.787 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:49.787 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:45:49.788 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:49.788 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:49.788 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:49.788 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:45:49.791 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:49.792 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:49.792 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:49.792 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:45:49.792 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:49.793 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:45:49.793 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:45:49.793 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:45:49.793 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:49.793 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:45:49.795 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.875533_162.1 2021-09-02T07:45:54.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:45:54.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:45:54.797 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:45:54.884 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:45:54.884 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:45:55.431 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:45:55.431 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:45:55.431 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:45:55.432 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:45:55.432 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:45:55.432 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:45:55.432 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:55.433 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:55.433 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:45:55.433 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:55.433 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:45:55.433 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:55.434 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:55.434 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:45:55.434 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:55.434 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:45:55.435 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:55.435 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:45:55.435 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:55.435 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:55.435 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:55.436 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:55.436 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:45:55.436 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:55.436 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:55.437 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.869904_161.1", 2021-09-02T07:45:55.437 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:55.438 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:55.438 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:55.438 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.869904Z", 2021-09-02T07:45:55.439 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:55.439 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:55.439 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:55.439 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:55.440 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:45:55.440 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:55.440 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:55.441 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:45:55.441 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:55.441 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:55.441 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:55.441 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:45:55.442 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:55.445 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:55.445 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:55.446 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:45:55.446 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:45:55.446 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:45:55.446 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:45:55.447 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:45:55.447 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:45:55.447 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:45:55.447 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:45:55.448 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:45:55.448 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:45:55.449 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:55.449 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:55.449 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:45:55.450 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:45:55.450 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:45:55.451 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:45:55.451 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:45:55.451 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:45:55.452 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:45:55.453 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.869904_161.1 2021-09-02T07:46:00.455 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:00.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:00.456 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:46:00.534 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:46:00.535 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:46:00.795 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:46:00.796 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:46:00.796 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:46:00.796 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:46:00.797 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:46:00.797 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:46:00.797 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:00.797 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:00.798 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:46:00.798 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:00.798 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:46:00.799 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:00.799 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:00.799 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:46:00.799 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:00.800 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:46:00.800 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:00.800 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:46:00.800 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:00.800 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:00.801 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:00.801 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:00.801 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:46:00.801 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:00.802 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:00.802 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.835437_156.1", 2021-09-02T07:46:00.802 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:00.802 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:00.802 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:00.803 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.835437Z", 2021-09-02T07:46:00.803 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:00.803 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:00.803 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:00.804 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:00.804 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:46:00.804 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:00.804 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:00.804 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:00.805 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:00.805 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:00.805 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:00.805 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:00.806 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:00.806 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:00.806 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:00.806 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:00.806 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:46:00.807 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:00.807 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:00.807 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:00.807 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:00.808 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:00.808 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:00.808 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:00.808 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:00.808 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:00.809 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:00.809 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:46:00.809 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:00.810 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:46:00.810 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:46:00.810 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:46:00.810 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:00.811 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:46:00.811 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.835437_156.1 2021-09-02T07:46:00.948 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2021-09-02T07:46:00.949+0000 7f04206f5700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-02T07:46:05.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:05.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:05.814 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:46:05.895 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:46:05.896 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:46:06.483 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:46:06.487 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:46:06.487 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:46:06.488 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:46:06.488 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:46:06.489 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:46:06.489 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:06.490 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:06.508 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:46:06.509 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:06.509 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:46:06.509 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:06.509 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:06.510 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:46:06.510 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:06.510 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:46:06.510 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:06.511 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:46:06.511 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:06.511 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:06.511 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:06.511 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:06.512 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:46:06.516 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:06.517 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:06.517 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568129.951314_211.1", 2021-09-02T07:46:06.517 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:06.517 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:06.518 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:06.518 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:29.951314Z", 2021-09-02T07:46:06.518 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:06.518 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:06.519 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:06.519 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:06.519 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:46:06.519 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:06.520 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:06.520 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:06.520 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:06.520 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:06.520 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:06.521 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:06.521 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:06.521 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:06.521 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:06.522 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:06.522 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:46:06.522 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:06.544 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:06.544 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:06.545 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:06.545 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:06.545 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:06.545 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:06.546 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:06.546 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:06.546 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:06.547 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:46:06.547 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:06.548 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:46:06.548 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:46:06.548 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:46:06.549 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:06.549 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:46:06.565 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568129.951314_211.1 2021-09-02T07:46:11.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:11.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:11.574 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:46:11.649 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:46:11.650 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:46:11.886 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:46:11.886 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:46:11.886 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:46:11.887 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:46:11.887 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:46:11.887 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:46:11.887 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:11.887 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:11.888 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:46:11.888 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:11.888 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:46:11.888 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:11.889 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:11.889 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:46:11.889 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:11.889 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:46:11.889 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:11.890 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:46:11.890 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:11.890 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:11.890 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:11.891 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:11.891 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:46:11.891 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:11.892 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:11.892 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.911309_165.1", 2021-09-02T07:46:11.892 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:11.893 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:11.893 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:11.893 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.911309Z", 2021-09-02T07:46:11.893 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:11.893 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:11.894 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:11.894 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:11.894 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:46:11.894 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:11.895 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:11.895 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:11.895 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:11.895 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:11.895 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:11.896 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:11.896 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:11.896 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:11.896 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:11.897 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:11.897 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:46:11.897 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:11.897 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:11.897 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:11.898 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:11.898 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:11.898 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:11.898 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:11.899 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:11.899 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:11.899 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:11.899 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:46:11.899 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:11.900 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:46:11.900 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:46:11.900 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:46:11.900 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:11.900 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:46:11.904 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.911309_165.1 2021-09-02T07:46:13.936 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:46:13.937+0000 7f968e196700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-02T07:46:16.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:16.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:16.911 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:46:16.999 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:46:16.999 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:46:17.111 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:46:17.111 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:46:17.111 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:46:17.112 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:46:17.112 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:46:17.112 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:46:17.112 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:17.113 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:17.113 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:46:17.114 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:17.114 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:46:17.114 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:17.114 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:17.115 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:46:17.115 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:17.115 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:46:17.115 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:17.116 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:46:17.116 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:17.116 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:17.116 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:17.117 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:17.117 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:46:17.117 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:17.117 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:17.117 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.862512_159.1", 2021-09-02T07:46:17.118 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:17.118 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:17.118 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:17.119 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.862512Z", 2021-09-02T07:46:17.119 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:17.119 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:17.119 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:17.120 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:17.120 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:46:17.120 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:17.121 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:17.121 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:17.121 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:17.121 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:17.122 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:17.122 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:17.122 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:17.122 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:17.123 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:17.123 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:17.123 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:46:17.124 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:17.124 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:17.124 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:17.124 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:17.125 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:17.125 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:17.125 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:17.125 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:17.126 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:17.126 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:17.126 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:46:17.126 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:17.127 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:46:17.127 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:46:17.127 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:46:17.127 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:17.127 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:46:17.129 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.862512_159.1 2021-09-02T07:46:22.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:22.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:22.132 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:46:22.221 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:46:22.221 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:46:22.463 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:46:22.464 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:46:22.464 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:46:22.464 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:46:22.464 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:46:22.464 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:46:22.465 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:22.465 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:22.465 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:46:22.465 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:22.466 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:46:22.466 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:22.467 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:22.467 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:46:22.468 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:22.468 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:46:22.468 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:22.469 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:46:22.469 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:22.470 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:22.470 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:22.471 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:22.472 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:46:22.473 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:22.473 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:22.473 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.129118_179.1", 2021-09-02T07:46:22.473 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:22.474 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:22.474 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:22.474 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.129118Z", 2021-09-02T07:46:22.474 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:22.475 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:22.475 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:22.475 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:22.475 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:46:22.475 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:22.476 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:22.476 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:22.476 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:22.476 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:22.477 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:22.477 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:22.477 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:22.477 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:22.477 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:22.478 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:22.478 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:46:22.478 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:22.478 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:22.479 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:22.479 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:22.479 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:22.479 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:22.480 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:22.480 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:22.480 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:22.480 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:22.481 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:46:22.481 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:22.481 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:46:22.481 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:46:22.482 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:46:22.482 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:22.482 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:46:22.483 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568127.129118_179.1 2021-09-02T07:46:27.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:27.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:27.486 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:46:27.568 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:46:27.569 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:46:27.925 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:46:27.926 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:46:27.926 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:46:27.926 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:46:27.926 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:46:27.927 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:46:27.927 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:27.927 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:27.927 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:46:27.928 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:27.928 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:46:27.928 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:27.928 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:27.929 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:46:27.929 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:27.929 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:46:27.929 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:27.930 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:46:27.930 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:27.931 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:27.931 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:27.931 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:27.932 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:46:27.932 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:27.933 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:27.933 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.867814_160.1", 2021-09-02T07:46:27.933 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:27.934 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:27.934 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:27.934 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.867814Z", 2021-09-02T07:46:27.934 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:27.935 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:27.935 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:27.935 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:27.936 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:46:27.936 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:27.936 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:27.936 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:27.937 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:27.937 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:27.937 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:27.937 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:27.938 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:27.938 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:27.938 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:27.938 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:27.939 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:46:27.939 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:27.939 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:27.939 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:27.940 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:27.940 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:27.940 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:27.940 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:27.941 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:27.941 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:27.942 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:27.942 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:46:27.942 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:27.942 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:46:27.943 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:46:27.943 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:46:27.943 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:27.943 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:46:27.945 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.867814_160.1 2021-09-02T07:46:32.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:32.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:32.949 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:46:33.035 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:46:33.035 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:46:33.402 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:46:33.402 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:46:33.402 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:46:33.403 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:46:33.404 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:46:33.404 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:46:33.404 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:33.404 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:33.405 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:46:33.405 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:33.405 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:46:33.405 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:33.406 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:33.406 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:46:33.406 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:33.406 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:46:33.407 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:33.407 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:46:33.407 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:33.408 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:33.408 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:33.408 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:33.408 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:46:33.409 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:33.409 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:33.409 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.911309_165.1", 2021-09-02T07:46:33.409 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:33.410 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:33.410 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:33.410 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.911309Z", 2021-09-02T07:46:33.410 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:33.411 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:33.411 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:33.411 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:33.411 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:46:33.411 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:33.412 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:33.412 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:33.412 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:33.412 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:33.413 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:33.413 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:33.413 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:33.414 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:33.414 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:33.414 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:33.415 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:46:33.415 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:33.415 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:33.415 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:33.416 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:33.416 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:33.417 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:33.417 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:33.418 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:33.419 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:33.419 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:33.419 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:46:33.419 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:33.420 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:46:33.420 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:46:33.420 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:46:33.420 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:33.421 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:46:33.422 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.911309_165.1 2021-09-02T07:46:38.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:38.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:38.425 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:46:38.505 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:46:38.507 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:46:38.677 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:46:38.677 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:46:38.678 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:46:38.678 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:46:38.678 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:46:38.678 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:46:38.678 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:38.679 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:38.679 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:46:38.679 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:38.679 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:46:38.680 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:38.680 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:38.680 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:46:38.680 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:38.681 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:46:38.681 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:38.681 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:46:38.681 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:38.681 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:38.682 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:38.682 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:38.682 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:46:38.682 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:38.683 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:38.683 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.911309_165.1", 2021-09-02T07:46:38.683 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:38.683 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:38.684 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:38.684 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.911309Z", 2021-09-02T07:46:38.684 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:38.684 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:38.685 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:38.685 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:38.686 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:46:38.686 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:38.686 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:38.686 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:38.687 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:38.687 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:38.687 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:38.687 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:38.687 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:38.688 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:38.688 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:38.688 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:38.688 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:46:38.689 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:38.689 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:38.689 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:38.689 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:38.690 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:38.690 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:38.690 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:38.690 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:38.690 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:38.691 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:38.691 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:46:38.691 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:38.691 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:46:38.692 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:46:38.692 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:46:38.692 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:38.692 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:46:38.694 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.911309_165.1 2021-09-02T07:46:43.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:43.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:43.696 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:46:43.779 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:46:43.779 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:46:44.001 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:46:44.001 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:46:44.002 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:46:44.002 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:46:44.002 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:46:44.002 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:46:44.003 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:44.003 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:44.003 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:46:44.003 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:44.003 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:46:44.004 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:44.004 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:44.004 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:46:44.004 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:44.005 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:46:44.005 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:44.005 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:46:44.005 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:44.006 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:44.006 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:44.006 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:44.006 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:46:44.006 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:44.007 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:44.007 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.932719_168.1", 2021-09-02T07:46:44.007 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:44.007 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:44.008 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:44.008 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.932719Z", 2021-09-02T07:46:44.008 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:44.008 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:44.008 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:44.009 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:44.009 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:46:44.009 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:44.010 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:44.010 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:44.011 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:44.011 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:44.011 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:44.012 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:44.012 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:44.012 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:44.012 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:44.013 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:44.013 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:46:44.013 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:44.014 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:44.014 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:44.015 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:44.015 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:44.015 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:44.016 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:44.016 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:44.016 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:44.016 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:44.016 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:46:44.017 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:44.017 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:46:44.017 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:46:44.017 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:46:44.018 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:44.018 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:46:44.020 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.932719_168.1 2021-09-02T07:46:49.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:49.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:49.023 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:46:49.112 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:46:49.113 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:46:49.467 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:46:49.468 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:46:49.468 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:46:49.468 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:46:49.468 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:46:49.469 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:46:49.469 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:49.469 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:49.469 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:46:49.470 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:49.470 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:46:49.470 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:49.471 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:49.471 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:46:49.472 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:49.472 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:46:49.472 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:49.472 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:46:49.473 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:49.473 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:49.473 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:49.473 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:49.474 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:46:49.474 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:49.474 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:49.474 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.869904_161.1", 2021-09-02T07:46:49.475 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:49.475 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:49.475 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:49.475 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.869904Z", 2021-09-02T07:46:49.476 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:49.476 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:49.476 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:49.476 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:49.477 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:46:49.477 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:49.477 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:49.478 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:49.478 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:49.478 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:49.478 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:49.479 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:49.479 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:49.479 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:49.479 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:49.479 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:49.480 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:46:49.480 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:49.480 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:49.480 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:49.481 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:49.481 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:49.481 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:49.482 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:49.482 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:49.482 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:49.483 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:49.483 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:46:49.483 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:49.483 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:46:49.484 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:46:49.484 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:46:49.484 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:49.484 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:46:49.485 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.869904_161.1 2021-09-02T07:46:54.486 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:54.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:54.488 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:46:54.582 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:46:54.582 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:46:54.835 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:46:54.836 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:46:54.836 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:46:54.836 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:46:54.836 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:46:54.837 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:46:54.837 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:54.837 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:54.837 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:46:54.838 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:54.838 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:46:54.838 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:54.838 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:54.838 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:46:54.839 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:54.839 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:46:54.839 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:54.840 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:46:54.840 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:54.840 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:54.841 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:54.841 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:54.841 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:46:54.841 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:54.842 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:54.842 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T07:46:54.842 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:54.842 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:54.843 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:54.843 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T07:46:54.843 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:54.843 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:54.846 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:54.846 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:54.846 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:46:54.847 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:54.847 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:54.847 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:54.848 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:54.848 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:54.848 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:54.850 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:54.850 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:54.850 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:54.851 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:54.851 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:46:54.851 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:46:54.854 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:46:54.855 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:46:54.856 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:46:54.856 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:46:54.856 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:46:54.856 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:46:54.857 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:46:54.857 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:46:54.857 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:54.857 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:54.857 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:46:54.858 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:46:54.858 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:46:54.858 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:46:54.859 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:46:54.859 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:46:54.859 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:46:54.861 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.906521_163.1 2021-09-02T07:46:59.863 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:59.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:46:59.865 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:46:59.943 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:46:59.944 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:47:00.118 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:47:00.119 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:47:00.119 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:47:00.120 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:47:00.121 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:47:00.121 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:47:00.121 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:00.121 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:00.122 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:47:00.122 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:00.122 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:47:00.122 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:00.123 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:00.123 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:47:00.123 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:00.123 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:47:00.124 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:00.124 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:47:00.124 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:00.124 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:00.124 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:00.125 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:00.125 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:47:00.125 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:00.125 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:00.126 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.914475_166.1", 2021-09-02T07:47:00.126 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:00.126 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:00.126 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:00.127 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.914475Z", 2021-09-02T07:47:00.127 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:00.127 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:00.127 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:00.128 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:00.128 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:47:00.128 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:00.128 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:00.129 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:00.129 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:00.129 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:00.130 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:00.130 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:00.130 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:00.130 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:00.131 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:00.131 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:00.131 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:47:00.131 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:00.131 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:00.132 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:00.132 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:00.132 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:00.132 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:00.132 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:00.133 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:00.133 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:00.133 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:00.133 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:47:00.134 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:00.134 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:47:00.134 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:47:00.134 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:47:00.134 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:00.135 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:47:00.136 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.914475_166.1 2021-09-02T07:47:05.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:05.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:05.140 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:47:05.227 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:47:05.227 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:47:05.362 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:47:05.362 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:47:05.362 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:47:05.363 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:47:05.363 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:47:05.363 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:47:05.363 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:05.364 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:05.364 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:47:05.364 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:05.364 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:47:05.365 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:05.366 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:05.366 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:47:05.366 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:05.367 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:47:05.367 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:05.367 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:47:05.367 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:05.370 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:05.370 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:05.370 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:05.371 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:47:05.371 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:05.371 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:05.371 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T07:47:05.371 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:05.372 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:05.372 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:05.372 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T07:47:05.372 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:05.373 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:05.373 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:05.373 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:05.373 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:47:05.374 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:05.374 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:05.374 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:05.374 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:05.374 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:05.375 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:05.375 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:05.375 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:05.375 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:05.376 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:05.376 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:05.376 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:47:05.376 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:05.377 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:05.377 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:05.377 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:05.378 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:05.378 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:05.378 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:05.378 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:05.379 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:05.379 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:05.379 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:47:05.379 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:05.380 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:47:05.380 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:47:05.381 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:47:05.381 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:05.381 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:47:05.383 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568127.103991_175.1 2021-09-02T07:47:10.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:10.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:10.386 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:47:10.467 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:47:10.467 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:47:10.914 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:47:10.915 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:47:10.915 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:47:10.915 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:47:10.916 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:47:10.916 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:47:10.916 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:10.917 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:10.917 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:47:10.917 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:10.918 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:47:10.918 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:10.918 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:10.918 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:47:10.919 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:10.919 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:47:10.919 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:10.919 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:47:10.920 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:10.920 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:10.920 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:10.920 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:10.921 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:47:10.921 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:10.921 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:10.921 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T07:47:10.921 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:10.922 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:10.922 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:10.922 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T07:47:10.922 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:10.923 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:10.923 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:10.923 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:10.923 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:47:10.924 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:10.924 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:10.924 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:10.924 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:10.924 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:10.925 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:10.925 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:10.925 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:10.925 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:10.926 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:10.926 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:10.926 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:47:10.926 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:10.927 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:10.927 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:10.927 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:10.927 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:10.927 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:10.928 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:10.928 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:10.928 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:10.928 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:10.929 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:47:10.929 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:10.929 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:47:10.929 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:47:10.929 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:47:10.930 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:10.930 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:47:10.931 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.910082_164.1 2021-09-02T07:47:15.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:15.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:15.934 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:47:16.035 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:47:16.036 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:47:16.369 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:47:16.370 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:47:16.370 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:47:16.370 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:47:16.370 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:47:16.371 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:47:16.371 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:16.371 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:16.372 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:47:16.372 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:16.372 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:47:16.372 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:16.373 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:16.373 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:47:16.373 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:16.374 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:47:16.374 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:16.374 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:47:16.374 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:16.375 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:16.375 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:16.375 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:16.375 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:47:16.376 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:16.376 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:16.376 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T07:47:16.377 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:16.378 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:16.378 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:16.378 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T07:47:16.379 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:16.379 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:16.379 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:16.379 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:16.380 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:47:16.380 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:16.380 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:16.380 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:16.380 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:16.381 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:16.381 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:16.381 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:16.381 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:16.382 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:16.382 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:16.382 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:16.382 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:47:16.383 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:16.383 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:16.383 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:16.383 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:16.384 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:16.384 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:16.384 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:16.386 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:16.387 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:16.387 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:16.387 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:47:16.387 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:16.388 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:47:16.388 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:47:16.388 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:47:16.388 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:16.388 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:47:16.389 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.910082_164.1 2021-09-02T07:47:21.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:21.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:21.393 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:47:21.486 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:47:21.486 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:47:21.914 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:47:21.915 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:47:21.915 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:47:21.915 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:47:21.916 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:47:21.916 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:47:21.916 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:21.917 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:21.917 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:47:21.917 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:21.917 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:47:21.918 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:21.918 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:21.918 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:47:21.918 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:21.918 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:47:21.919 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:21.919 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:47:21.919 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:21.919 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:21.920 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:21.920 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:21.920 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:47:21.920 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:21.921 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:21.921 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.929348_167.1", 2021-09-02T07:47:21.921 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:21.921 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:21.922 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:21.922 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.929348Z", 2021-09-02T07:47:21.922 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:21.922 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:21.923 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:21.923 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:21.923 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:47:21.923 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:21.923 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:21.924 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:21.924 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:21.924 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:21.924 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:21.925 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:21.925 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:21.925 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:21.926 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:21.926 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:21.927 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:47:21.927 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:21.927 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:21.928 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:21.928 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:21.929 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:21.929 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:21.930 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:21.930 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:21.931 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:21.931 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:21.931 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:47:21.932 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:21.932 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:47:21.933 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:47:21.933 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:47:21.934 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:21.934 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:47:21.935 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.929348_167.1 2021-09-02T07:47:26.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:26.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:26.938 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:47:27.018 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:47:27.018 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:47:27.305 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:47:27.306 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:47:27.306 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:47:27.307 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:47:27.307 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:47:27.308 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:47:27.308 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:27.308 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:27.309 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:47:27.309 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:27.310 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:47:27.310 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:27.311 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:27.311 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:47:27.312 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:27.312 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:47:27.313 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:27.313 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:47:27.314 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:27.315 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:27.315 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:27.315 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:27.316 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:47:27.316 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:27.317 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:27.317 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.039849_171.1", 2021-09-02T07:47:27.319 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:27.319 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:27.320 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:27.321 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.039849Z", 2021-09-02T07:47:27.321 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:27.323 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:27.324 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:27.325 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:27.325 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:47:27.326 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:27.326 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:27.327 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:27.327 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:27.328 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:27.328 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:27.328 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:27.329 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:27.329 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:27.332 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:27.332 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:27.333 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:47:27.334 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:27.341 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:27.342 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:27.342 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:27.343 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:27.343 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:27.344 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:27.344 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:27.345 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:27.345 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:27.345 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:47:27.345 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:27.346 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:47:27.346 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:47:27.346 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:47:27.346 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:27.347 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:47:27.348 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568127.039849_171.1 2021-09-02T07:47:32.350 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:32.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:32.353 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:47:32.436 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:47:32.437 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:47:32.668 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:47:32.669 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:47:32.670 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:47:32.670 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:47:32.670 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:47:32.671 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:47:32.671 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:32.671 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:32.671 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:47:32.671 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:32.672 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:47:32.672 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:32.672 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:32.672 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:47:32.673 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:32.673 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:47:32.673 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:32.673 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:47:32.673 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:32.674 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:32.674 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:32.674 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:32.674 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:47:32.675 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:32.675 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:32.675 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T07:47:32.675 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:32.676 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:32.676 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:32.676 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T07:47:32.676 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:32.676 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:32.677 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:32.677 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:32.677 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:47:32.677 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:32.678 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:32.678 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:32.678 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:32.678 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:32.679 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:32.679 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:32.679 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:32.680 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:32.680 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:32.680 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:32.680 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:47:32.681 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:32.681 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:32.681 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:32.681 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:32.681 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:32.682 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:32.682 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:32.682 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:32.682 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:32.683 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:32.683 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:47:32.683 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:32.683 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:47:32.684 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:47:32.684 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:47:32.684 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:32.684 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:47:32.686 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568127.103991_175.1 2021-09-02T07:47:37.687 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:37.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:37.689 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:47:37.767 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:47:37.768 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:47:37.932 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:47:37.933 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:47:37.933 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:47:37.933 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:47:37.934 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:47:37.934 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:47:37.935 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:37.935 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:37.935 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:47:37.936 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:37.936 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:47:37.936 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:37.937 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:37.937 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:47:37.938 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:37.938 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:47:37.939 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:37.939 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:47:37.939 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:37.940 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:37.940 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:37.940 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:37.941 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:47:37.941 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:37.941 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:37.942 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T07:47:37.942 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:37.942 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:37.943 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:37.944 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T07:47:37.944 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:37.944 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:37.944 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:37.945 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:37.945 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:47:37.945 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:37.945 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:37.946 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:37.946 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:37.946 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:37.946 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:37.947 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:37.947 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:37.947 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:37.947 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:37.947 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:37.948 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:47:37.948 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:37.948 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:37.948 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:37.949 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:37.949 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:37.949 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:37.949 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:37.949 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:37.950 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:37.950 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:37.950 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:47:37.950 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:37.951 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:47:37.951 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:47:37.951 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:47:37.951 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:37.952 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:47:37.953 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568127.103991_175.1 2021-09-02T07:47:42.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:42.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:42.956 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:47:43.054 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:47:43.054 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:47:43.456 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:47:43.457 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:47:43.458 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:47:43.458 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:47:43.458 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:47:43.459 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:47:43.459 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:43.459 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:43.459 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:47:43.460 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:43.460 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:47:43.460 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:43.460 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:43.460 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:47:43.461 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:43.461 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:47:43.461 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:43.461 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:47:43.462 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:43.462 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:43.462 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:43.462 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:43.463 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:47:43.463 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:43.463 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:43.463 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.129118_179.1", 2021-09-02T07:47:43.463 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:43.464 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:43.464 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:43.464 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.129118Z", 2021-09-02T07:47:43.465 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:43.465 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:43.465 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:43.465 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:43.465 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:47:43.466 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:43.466 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:43.466 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:43.466 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:43.467 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:43.467 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:43.467 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:43.467 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:43.467 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:43.468 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:43.468 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:43.468 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:47:43.468 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:43.468 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:43.469 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:43.469 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:43.469 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:43.469 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:43.470 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:43.470 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:43.470 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:43.470 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:43.470 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:47:43.471 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:43.471 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:47:43.471 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:47:43.471 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:47:43.472 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:43.472 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:47:43.473 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568127.129118_179.1 2021-09-02T07:47:48.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:48.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:48.477 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:47:48.560 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:47:48.617 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:47:48.929 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:47:48.932 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:47:48.933 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:47:48.933 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:47:48.933 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:47:48.933 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:47:48.934 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:48.934 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:48.934 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:47:48.934 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:48.935 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:47:48.935 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:48.935 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:48.935 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:47:48.935 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:48.936 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:47:48.936 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:48.936 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:47:48.936 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:48.937 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:48.937 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:48.937 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:48.937 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:47:48.938 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:48.938 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:48.938 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T07:47:48.938 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:48.939 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:48.939 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:48.940 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T07:47:48.940 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:48.940 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:48.940 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:48.941 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:48.941 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:47:48.941 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:48.941 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:48.941 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:48.942 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:48.942 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:48.942 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:48.942 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:48.943 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:48.943 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:48.943 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:48.943 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:48.943 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:47:48.944 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:48.944 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:48.944 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:48.945 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:48.945 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:48.945 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:48.945 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:48.946 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:48.946 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:48.946 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:48.947 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:47:48.947 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:48.947 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:47:48.947 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:47:48.947 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:47:48.948 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:48.948 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:47:48.948 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.910082_164.1 2021-09-02T07:47:53.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:53.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:53.954 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:47:54.062 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:47:54.062 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:47:54.443 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:47:54.443 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:47:54.443 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:47:54.444 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:47:54.444 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:47:54.445 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:47:54.445 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:54.445 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:54.446 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:47:54.446 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:54.446 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:47:54.447 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:54.447 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:54.447 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:47:54.447 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:54.447 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:47:54.448 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:54.448 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:47:54.448 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:54.448 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:54.449 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:54.449 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:54.449 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:47:54.449 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:54.450 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:54.450 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.929348_167.1", 2021-09-02T07:47:54.450 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:54.450 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:54.450 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:54.451 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.929348Z", 2021-09-02T07:47:54.451 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:54.451 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:54.451 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:54.452 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:54.453 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:47:54.453 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:54.454 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:54.454 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:54.454 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:54.454 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:54.455 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:54.455 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:54.455 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:54.455 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:54.456 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:54.456 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:54.457 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:47:54.457 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:54.457 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:54.457 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:54.458 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:54.458 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:54.458 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:54.458 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:54.458 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:54.459 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:54.459 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:54.459 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:47:54.459 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:54.460 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:47:54.460 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:47:54.460 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:47:54.460 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:54.460 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:47:54.461 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.929348_167.1 2021-09-02T07:47:59.463 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:59.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:47:59.464 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:47:59.571 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:47:59.572 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:47:59.781 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:47:59.781 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:47:59.782 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:47:59.782 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:47:59.782 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:47:59.782 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:47:59.783 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:59.783 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:59.783 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:47:59.783 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:59.783 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:47:59.784 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:59.784 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:59.784 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:47:59.784 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:59.785 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:47:59.785 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:59.785 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:47:59.786 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:59.787 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:59.788 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:59.788 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:59.788 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:47:59.788 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:59.788 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:59.789 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.168223_183.1", 2021-09-02T07:47:59.789 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:59.789 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:59.790 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:59.790 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.168223Z", 2021-09-02T07:47:59.790 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:59.790 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:59.791 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:59.791 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:59.791 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:47:59.791 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:59.792 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:59.792 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:59.793 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:59.794 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:59.794 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:59.794 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:59.795 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:59.795 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:59.795 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:59.795 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:47:59.795 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:47:59.796 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:47:59.796 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:47:59.796 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:47:59.796 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:47:59.797 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:47:59.797 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:47:59.797 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:47:59.797 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:47:59.798 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:59.798 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:59.799 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:47:59.799 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:47:59.799 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:47:59.799 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:47:59.800 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:47:59.800 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:47:59.800 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:47:59.801 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568127.168223_183.1 2021-09-02T07:48:04.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:04.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:04.804 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:48:04.897 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:48:04.898 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:48:05.029 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:48:05.029 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:48:05.030 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:48:05.030 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:48:05.030 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:48:05.031 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:48:05.031 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:05.031 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:05.032 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:48:05.032 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:05.032 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:48:05.032 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:05.033 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:05.033 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:48:05.033 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:05.033 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:48:05.033 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:05.034 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:48:05.034 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:05.034 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:05.034 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:05.035 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:05.035 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:48:05.035 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:05.035 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:05.035 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T07:48:05.036 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:05.036 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:05.036 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:05.036 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T07:48:05.037 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:05.037 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:05.037 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:05.037 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:05.037 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:48:05.038 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:05.038 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:05.038 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:05.038 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:05.039 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:05.039 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:05.040 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:05.040 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:05.040 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:05.040 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:05.041 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:05.041 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:48:05.041 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:05.041 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:05.041 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:05.042 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:05.042 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:05.042 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:05.042 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:05.043 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:05.043 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:05.043 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:05.043 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:48:05.043 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:05.044 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:48:05.044 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:48:05.044 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:48:05.044 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:05.045 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:48:05.046 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.906521_163.1 2021-09-02T07:48:10.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:10.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:10.048 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:48:10.129 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:48:10.130 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:48:10.253 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:48:10.253 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:48:10.253 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:48:10.254 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:48:10.254 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:48:10.254 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:48:10.254 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:10.255 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:10.255 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:48:10.255 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:10.255 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:48:10.255 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:10.256 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:10.256 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:48:10.256 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:10.256 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:48:10.257 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:10.257 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:48:10.257 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:10.257 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:10.258 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:10.258 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:10.258 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:48:10.258 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:10.258 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:10.259 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.929348_167.1", 2021-09-02T07:48:10.259 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:10.259 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:10.259 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:10.260 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.929348Z", 2021-09-02T07:48:10.260 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:10.260 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:10.260 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:10.260 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:10.261 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:48:10.261 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:10.261 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:10.261 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:10.261 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:10.262 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:10.262 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:10.262 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:10.262 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:10.263 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:10.263 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:10.263 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:10.263 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:48:10.264 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:10.264 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:10.264 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:10.264 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:10.264 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:10.265 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:10.265 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:10.265 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:10.265 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:10.266 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:10.266 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:48:10.266 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:10.266 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:48:10.266 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:48:10.267 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:48:10.267 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:10.267 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:48:10.268 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.929348_167.1 2021-09-02T07:48:15.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:15.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:15.271 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:48:15.348 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:48:15.348 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:48:15.649 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:48:15.649 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:48:15.650 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:48:15.650 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:48:15.650 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:48:15.650 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:48:15.651 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:15.651 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:15.651 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:48:15.652 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:15.652 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:48:15.652 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:15.652 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:15.653 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:48:15.653 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:15.653 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:48:15.653 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:15.653 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:48:15.654 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:15.654 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:15.654 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:15.654 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:15.655 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:48:15.655 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:15.655 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:15.655 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T07:48:15.656 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:15.656 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:15.656 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:15.657 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T07:48:15.657 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:15.657 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:15.657 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:15.658 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:15.658 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:48:15.658 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:15.658 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:15.659 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:15.659 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:15.659 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:15.660 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:15.660 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:15.660 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:15.660 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:15.661 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:15.661 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:15.661 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:48:15.661 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:15.661 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:15.662 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:15.662 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:15.662 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:15.662 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:15.663 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:15.663 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:15.663 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:15.663 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:15.664 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:48:15.664 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:15.664 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:48:15.664 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:48:15.664 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:48:15.665 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:15.665 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:48:15.666 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568126.910082_164.1 2021-09-02T07:48:20.668 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:20.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:20.670 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:48:20.748 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:48:20.748 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:48:21.021 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:48:21.021 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:48:21.021 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:48:21.021 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:48:21.022 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:48:21.022 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:48:21.022 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:21.022 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:21.023 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:48:21.023 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:21.023 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:48:21.023 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:21.024 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:21.024 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:48:21.024 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:21.024 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:48:21.024 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:21.025 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:48:21.025 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:21.025 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:21.025 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:21.026 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:21.026 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:48:21.026 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:21.026 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:21.027 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T07:48:21.027 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:21.027 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:21.027 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:21.027 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T07:48:21.028 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:21.028 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:21.028 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:21.029 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:21.029 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:48:21.029 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:21.029 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:21.029 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:21.030 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:21.030 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:21.030 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:21.030 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:21.031 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:21.031 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:21.031 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:21.031 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:21.031 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:48:21.032 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:21.032 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:21.032 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:21.032 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:21.033 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:21.033 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:21.033 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:21.033 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:21.033 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:21.034 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:21.034 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:48:21.034 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:21.034 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:48:21.035 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:48:21.035 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:48:21.035 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:21.035 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:48:21.036 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568575.180614_247.1 target=1_1630568127.103991_175.1 2021-09-02T07:48:26.037 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... FAIL 2021-09-02T07:48:26.039 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-42 2021-09-02T07:48:26.074 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=efybpk-43 2021-09-02T07:48:26.206 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=efybpk-44 2021-09-02T07:48:26.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:26.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:26.396 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:48:26.406 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2021-09-02T07:48:26.407+0000 7f0430f16700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:48:26.407 INFO:tasks.rgw.c2.client.1.smithi197.stdout:2021-09-02T07:48:26.408+0000 7fad708af700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:48:26.407 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:48:26.408+0000 7f96a2dd9700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:48:26.440 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:48:26.440 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:48:26.493 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T07:48:26.497 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:48:26.499+0000 7f96a4ddd700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:48:26.498 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:48:26.500+0000 7f969b5ca700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:48:26.510 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:48:26.510 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:48:26.511 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:48:26.511 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:32:10.734248Z" 2021-09-02T07:48:26.511 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:48:26.511 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:48:26.512 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568906.235201_264.1", 2021-09-02T07:48:26.512 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:48:26.235201Z" 2021-09-02T07:48:26.512 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:48:26.512 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:48:26.513 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:48:26.513 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:48:26.513 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:48:26.513 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:48:26.514 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:48:26.514 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:48:26.514 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:48:26.515 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:48:26.516 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T07:48:26.516 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:26.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:26.518 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:48:26.598 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:48:26.598 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:48:26.991 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:48:26.991 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:48:26.991 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:48:26.992 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:48:26.992 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:48:26.992 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:48:26.992 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:26.993 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:26.993 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:48:26.993 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:26.993 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:48:26.993 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:26.994 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:26.994 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:48:26.994 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:26.994 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:48:26.995 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:26.995 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:48:26.995 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:26.995 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:26.996 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:26.996 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:26.996 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:48:26.996 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:26.996 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:26.997 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568906.062115_253.1", 2021-09-02T07:48:26.997 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:26.997 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:26.997 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:26.998 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:48:26.062115Z", 2021-09-02T07:48:26.998 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:26.998 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:26.998 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:26.998 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:26.999 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:48:26.999 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:26.999 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:26.999 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:27.000 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:27.000 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:27.000 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:27.000 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:27.000 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:27.001 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:27.001 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:27.001 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:27.001 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:48:27.002 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:27.002 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:27.002 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:27.002 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:27.002 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:27.003 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:27.003 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:27.003 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:27.003 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:27.004 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:27.004 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:48:27.004 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:27.004 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:48:27.004 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:48:27.005 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:48:27.005 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:27.005 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:48:27.006 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568906.062115_253.1 2021-09-02T07:48:27.607 INFO:tasks.rgw.c2.client.0.smithi197.stdout:2021-09-02T07:48:27.608+0000 7f96a15d6700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:48:27.607 INFO:tasks.rgw.c2.client.1.smithi197.stdout:2021-09-02T07:48:27.608+0000 7fad6d8a9700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:48:27.608 INFO:tasks.rgw.c2.client.1.smithi197.stdout:2021-09-02T07:48:27.608+0000 7fad6e0aa700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:48:32.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:32.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:32.010 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:48:32.081 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:48:32.082 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:48:32.194 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:48:32.195 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:48:32.195 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:48:32.195 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:48:32.196 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:48:32.196 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:48:32.196 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:32.196 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:32.196 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:48:32.197 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:32.197 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:48:32.197 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:32.197 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:32.198 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:48:32.198 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:32.198 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:48:32.198 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:32.199 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:48:32.199 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:32.199 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:32.199 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:32.200 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:32.200 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:48:32.200 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:32.200 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:32.200 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568906.235201_264.1", 2021-09-02T07:48:32.201 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:32.201 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:32.201 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:32.201 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:48:26.235201Z", 2021-09-02T07:48:32.202 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:32.202 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:32.202 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:32.202 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:32.202 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:48:32.203 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:32.203 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:32.203 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:32.203 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:32.204 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:32.204 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:32.204 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:32.204 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:32.204 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:32.205 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:32.205 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:32.205 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:48:32.205 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:32.206 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:32.206 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:32.206 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:32.206 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:32.206 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:32.207 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:32.207 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:32.207 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:32.207 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:32.208 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:48:32.208 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:32.208 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:48:32.208 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:48:32.208 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:48:32.209 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:32.209 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:48:32.210 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T07:48:32.211 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T07:48:32.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:32.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:32.212 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:48:32.259 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:48:32.260 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:48:32.336 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:48:32.337 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:48:32.337 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:48:32.337 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:48:32.337 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:48:32.338 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:48:32.338 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:48:32.338 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:48:32.338 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:48:32.338 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:48:32.339 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:48:32.339 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:48:32.339 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:48:32.339 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:48:32.340 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:48:32.340 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:48:32.340 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:48:32.340 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:48:32.341 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:48:32.341 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:48:32.341 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:48:32.341 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:48:32.341 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:48:32.342 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:48:32.342 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:48:32.342 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630568906.235201_264.1", 2021-09-02T07:48:32.342 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:48:32.343 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:48:32.343 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:48:32.343 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:48:26.235201Z", 2021-09-02T07:48:32.343 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:48:32.343 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:48:32.344 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:48:32.344 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:48:32.344 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:48:32.344 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:48:32.345 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:48:32.345 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:48:32.345 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:48:32.346 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:48:32.346 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:48:32.346 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:48:32.346 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:48:32.346 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:48:32.347 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:48:32.347 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:48:32.347 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:48:32.347 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:48:32.348 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:48:32.348 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:48:32.348 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:48:32.348 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:48:32.348 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:48:32.349 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:48:32.349 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:48:32.349 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:48:32.349 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:48:32.350 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:48:32.350 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:48:32.350 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:48:32.351 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:48:32.351 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:48:32.351 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:48:32.351 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:48:32.353 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T07:48:32.353 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T07:48:32.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:32.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:32.355 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:48:32.410 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:48:32.410 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:48:32.800 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:48:32.800 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:48:32.801 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:48:32.801 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:48:32.801 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:48:32.801 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:48:32.802 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:32.802 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:32.802 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:48:32.802 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:32.803 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:48:32.803 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:32.803 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:32.803 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:48:32.804 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:32.804 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:48:32.804 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:32.804 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:48:32.805 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:32.805 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:32.805 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:32.805 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:32.806 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:48:32.806 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:32.806 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:32.806 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.867814_160.1", 2021-09-02T07:48:32.806 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:32.807 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:32.807 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:32.807 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.867814Z", 2021-09-02T07:48:32.807 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:32.808 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:32.808 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:32.808 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:32.808 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:48:32.809 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:32.809 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:32.809 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:32.809 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:32.809 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:32.810 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:32.810 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:32.811 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:32.811 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:32.811 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:32.812 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:32.812 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:48:32.812 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:32.812 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:32.813 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:32.813 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:32.813 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:32.813 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:32.814 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:32.814 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:32.814 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:32.814 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:32.815 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:48:32.815 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:32.815 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:48:32.815 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:48:32.816 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:48:32.816 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:32.816 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:48:32.817 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.867814_160.1 2021-09-02T07:48:37.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:37.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:37.820 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:48:37.907 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:48:37.908 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:48:38.020 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:48:38.020 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:48:38.020 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:48:38.021 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:48:38.021 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:48:38.021 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:48:38.021 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:38.022 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:38.022 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:48:38.022 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:38.022 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:48:38.022 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:38.023 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:38.023 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:48:38.023 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:38.023 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:48:38.024 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:38.024 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:48:38.024 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:38.024 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:38.024 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:38.025 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:38.025 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:48:38.025 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:38.025 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:38.026 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.129118_179.1", 2021-09-02T07:48:38.026 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:38.026 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:38.026 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:38.026 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.129118Z", 2021-09-02T07:48:38.027 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:38.027 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:38.027 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:38.027 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:38.028 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:48:38.028 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:38.028 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:38.028 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:38.029 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:38.029 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:38.029 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:38.029 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:38.029 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:38.030 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:38.030 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:38.030 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:38.030 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:48:38.031 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:38.031 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:38.031 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:38.031 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:38.031 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:38.032 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:38.032 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:38.032 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:38.032 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:38.033 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:38.033 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:48:38.033 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:38.033 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:48:38.033 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:48:38.034 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:48:38.034 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:38.034 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:48:38.035 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568127.129118_179.1 2021-09-02T07:48:43.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:43.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:43.038 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:48:43.113 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:48:43.113 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:48:43.666 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:48:43.666 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:48:43.667 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:48:43.667 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:48:43.667 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:48:43.667 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:48:43.668 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:43.668 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:43.668 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:48:43.668 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:43.668 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:48:43.669 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:43.669 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:43.669 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:48:43.669 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:43.670 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:48:43.670 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:43.670 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:48:43.670 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:43.671 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:43.671 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:43.671 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:43.671 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:48:43.671 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:43.672 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:43.672 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T07:48:43.672 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:43.672 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:43.673 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:43.673 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T07:48:43.673 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:43.673 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:43.674 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:43.674 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:43.674 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:48:43.674 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:43.675 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:43.675 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:43.675 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:43.675 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:43.675 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:43.676 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:43.676 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:43.676 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:43.676 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:43.677 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:43.677 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:48:43.677 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:43.677 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:43.678 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:43.678 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:43.678 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:43.678 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:43.679 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:43.679 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:43.679 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:43.679 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:43.679 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:48:43.680 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:43.680 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:48:43.680 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:48:43.680 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:48:43.681 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:43.681 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:48:43.682 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.906521_163.1 2021-09-02T07:48:48.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:48.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:48.685 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:48:48.744 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:48:48.744 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:48:49.024 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:48:49.025 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:48:49.025 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:48:49.025 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:48:49.026 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:48:49.026 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:48:49.026 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:49.026 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:49.026 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:48:49.027 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:49.027 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:48:49.027 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:49.027 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:49.028 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:48:49.028 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:49.028 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:48:49.028 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:49.029 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:48:49.029 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:49.029 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:49.029 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:49.029 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:49.030 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:48:49.030 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:49.030 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:49.030 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T07:48:49.031 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:49.031 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:49.031 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:49.032 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T07:48:49.032 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:49.032 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:49.032 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:49.033 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:49.033 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:48:49.033 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:49.033 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:49.034 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:49.034 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:49.034 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:49.034 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:49.034 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:49.035 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:49.035 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:49.035 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:49.035 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:49.036 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:48:49.036 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:49.036 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:49.036 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:49.036 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:49.037 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:49.037 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:49.037 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:49.037 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:49.038 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:49.038 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:49.038 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:48:49.038 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:49.038 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:48:49.039 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:48:49.039 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:48:49.039 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:49.039 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:48:49.041 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.875533_162.1 2021-09-02T07:48:54.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:54.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:54.043 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:48:54.090 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:48:54.090 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:48:54.165 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:48:54.166 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:48:54.166 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:48:54.166 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:48:54.166 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:48:54.167 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:48:54.167 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:54.167 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:54.167 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:48:54.167 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:54.168 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:48:54.168 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:54.168 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:54.168 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:48:54.168 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:54.169 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:48:54.169 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:54.169 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:48:54.169 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:54.170 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:54.170 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:54.170 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:54.170 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:48:54.170 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:54.171 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:54.171 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.867814_160.1", 2021-09-02T07:48:54.171 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:54.171 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:54.172 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:54.172 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.867814Z", 2021-09-02T07:48:54.172 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:54.172 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:54.172 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:54.173 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:54.173 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:48:54.173 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:54.174 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:54.174 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:54.174 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:54.174 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:54.174 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:54.175 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:54.175 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:54.175 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:54.175 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:54.176 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:54.176 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:48:54.176 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:54.176 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:54.176 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:54.177 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:54.177 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:54.177 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:54.177 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:54.178 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:54.178 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:54.178 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:54.178 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:48:54.178 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:54.179 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:48:54.179 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:48:54.179 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:48:54.179 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:54.179 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:48:54.181 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.867814_160.1 2021-09-02T07:48:59.182 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:59.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:48:59.184 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:48:59.227 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:48:59.227 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:48:59.303 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:48:59.303 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:48:59.304 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:48:59.304 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:48:59.304 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:48:59.304 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:48:59.305 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:59.305 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:59.305 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:48:59.305 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:59.305 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:48:59.306 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:59.306 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:59.306 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:48:59.306 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:59.307 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:48:59.307 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:59.307 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:48:59.307 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:59.307 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:59.308 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:59.308 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:59.308 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:48:59.308 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:59.309 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:59.309 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T07:48:59.309 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:59.309 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:59.309 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:59.310 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T07:48:59.310 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:59.310 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:59.310 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:59.311 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:59.311 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:48:59.311 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:59.311 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:59.312 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:59.312 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:59.312 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:59.312 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:59.312 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:59.313 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:59.313 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:59.313 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:59.313 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:48:59.314 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:48:59.314 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:48:59.314 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:48:59.314 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:48:59.314 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:48:59.315 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:48:59.315 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:48:59.315 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:48:59.315 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:48:59.316 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:59.316 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:59.316 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:48:59.316 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:48:59.317 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:48:59.317 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:48:59.317 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:48:59.317 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:48:59.318 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:48:59.319 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568127.103991_175.1 2021-09-02T07:49:04.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:04.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:04.322 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:49:04.398 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:49:04.398 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:49:04.538 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:49:04.538 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:49:04.538 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:49:04.539 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:49:04.539 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:49:04.539 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:49:04.539 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:04.540 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:04.540 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:49:04.540 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:04.540 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:49:04.540 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:04.541 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:04.541 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:49:04.541 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:04.541 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:49:04.542 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:04.542 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:49:04.542 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:04.542 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:04.542 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:04.543 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:04.543 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:49:04.543 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:04.543 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:04.544 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T07:49:04.544 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:04.544 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:04.544 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:04.545 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T07:49:04.545 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:04.545 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:04.545 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:04.545 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:04.546 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:49:04.546 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:04.546 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:04.546 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:04.547 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:04.547 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:04.547 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:04.548 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:04.548 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:04.548 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:04.548 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:04.549 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:04.549 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:49:04.549 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:04.549 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:04.550 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:04.550 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:04.550 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:04.550 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:04.550 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:04.551 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:04.551 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:04.551 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:04.551 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:49:04.551 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:04.552 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:49:04.552 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:49:04.552 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:49:04.552 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:04.553 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:49:04.554 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.910082_164.1 2021-09-02T07:49:09.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:09.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:09.556 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:49:09.604 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:49:09.605 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:49:09.735 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:49:09.736 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:49:09.736 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:49:09.736 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:49:09.737 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:49:09.737 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:49:09.737 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:09.737 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:09.738 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:49:09.738 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:09.738 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:49:09.738 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:09.739 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:09.739 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:49:09.739 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:09.739 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:49:09.740 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:09.740 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:49:09.740 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:09.740 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:09.740 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:09.741 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:09.741 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:49:09.741 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:09.742 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:09.742 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.862512_159.1", 2021-09-02T07:49:09.742 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:09.742 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:09.743 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:09.743 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.862512Z", 2021-09-02T07:49:09.743 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:09.744 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:09.744 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:09.744 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:09.744 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:49:09.744 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:09.745 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:09.745 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:09.745 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:09.745 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:09.746 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:09.746 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:09.746 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:09.746 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:09.747 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:09.747 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:09.747 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:49:09.747 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:09.747 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:09.748 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:09.748 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:09.748 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:09.748 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:09.749 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:09.749 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:09.749 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:09.749 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:09.750 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:49:09.750 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:09.750 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:49:09.750 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:49:09.751 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:49:09.751 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:09.751 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:49:09.753 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.862512_159.1 2021-09-02T07:49:14.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:14.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:14.756 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:49:14.797 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:49:14.798 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:49:15.271 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:49:15.271 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:49:15.272 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:49:15.272 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:49:15.272 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:49:15.272 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:49:15.273 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:15.273 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:15.273 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:49:15.273 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:15.274 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:49:15.274 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:15.274 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:15.274 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:49:15.275 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:15.275 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:49:15.275 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:15.275 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:49:15.276 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:15.276 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:15.276 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:15.276 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:15.276 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:49:15.277 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:15.277 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:15.277 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T07:49:15.277 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:15.278 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:15.278 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:15.278 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T07:49:15.278 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:15.279 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:15.279 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:15.279 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:15.279 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:49:15.279 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:15.280 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:15.280 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:15.280 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:15.280 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:15.281 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:15.281 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:15.281 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:15.281 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:15.281 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:15.282 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:15.282 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:49:15.282 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:15.282 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:15.283 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:15.283 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:15.283 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:15.283 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:15.283 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:15.284 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:15.284 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:15.284 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:15.284 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:49:15.285 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:15.285 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:49:15.285 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:49:15.285 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:49:15.285 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:15.286 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:49:15.287 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568127.103991_175.1 2021-09-02T07:49:20.289 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:20.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:20.291 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:49:20.359 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:49:20.360 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:49:21.003 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:49:21.003 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:49:21.004 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:49:21.004 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:49:21.004 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:49:21.004 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:49:21.005 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:21.005 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:21.006 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:49:21.006 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:21.007 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:49:21.007 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:21.007 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:21.007 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:49:21.008 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:21.008 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:49:21.009 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:21.009 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:49:21.009 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:21.010 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:21.010 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:21.010 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:21.011 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:49:21.011 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:21.011 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:21.012 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T07:49:21.012 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:21.012 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:21.012 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:21.013 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T07:49:21.013 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:21.013 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:21.013 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:21.014 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:21.014 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:49:21.014 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:21.014 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:21.015 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:21.015 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:21.015 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:21.015 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:21.016 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:21.016 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:21.016 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:21.016 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:21.016 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:21.017 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:49:21.017 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:21.017 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:21.018 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:21.018 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:21.018 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:21.018 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:21.019 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:21.019 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:21.019 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:21.019 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:21.020 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:49:21.020 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:21.020 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:49:21.020 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:49:21.020 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:49:21.021 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:21.021 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:49:21.022 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568127.103991_175.1 2021-09-02T07:49:26.023 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:26.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:26.025 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:49:26.075 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:49:26.076 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:49:26.323 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:49:26.324 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:49:26.324 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:49:26.325 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:49:26.325 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:49:26.325 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:49:26.325 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:26.326 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:26.326 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:49:26.326 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:26.327 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:49:26.327 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:26.327 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:26.327 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:49:26.328 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:26.328 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:49:26.328 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:26.328 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:49:26.329 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:26.329 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:26.329 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:26.329 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:26.330 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:49:26.330 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:26.330 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:26.330 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.867814_160.1", 2021-09-02T07:49:26.331 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:26.331 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:26.331 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:26.331 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.867814Z", 2021-09-02T07:49:26.332 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:26.332 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:26.332 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:26.332 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:26.332 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:49:26.333 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:26.333 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:26.333 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:26.334 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:26.335 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:26.335 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:26.335 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:26.335 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:26.335 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:26.336 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:26.336 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:26.336 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:49:26.336 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:26.337 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:26.337 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:26.337 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:26.337 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:26.338 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:26.338 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:26.338 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:26.338 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:26.338 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:26.339 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:49:26.339 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:26.339 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:49:26.339 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:49:26.340 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:49:26.340 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:26.340 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:49:26.341 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.867814_160.1 2021-09-02T07:49:31.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:31.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:31.344 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:49:31.393 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:49:31.393 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:49:31.470 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:49:31.471 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:49:31.471 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:49:31.471 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:49:31.472 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:49:31.472 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:49:31.472 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:31.472 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:31.473 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:49:31.473 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:31.473 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:49:31.473 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:31.473 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:31.474 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:49:31.474 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:31.474 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:49:31.474 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:31.475 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:49:31.475 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:31.475 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:31.475 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:31.476 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:31.476 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:49:31.476 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:31.476 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:31.476 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.869904_161.1", 2021-09-02T07:49:31.477 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:31.477 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:31.477 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:31.478 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.869904Z", 2021-09-02T07:49:31.478 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:31.478 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:31.478 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:31.479 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:31.479 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:49:31.479 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:31.480 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:31.480 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:31.481 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:31.481 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:31.481 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:31.482 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:31.482 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:31.482 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:31.482 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:31.483 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:31.483 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:49:31.483 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:31.483 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:31.483 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:31.484 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:31.484 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:31.484 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:31.484 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:31.485 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:31.485 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:31.485 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:31.485 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:49:31.485 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:31.486 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:49:31.486 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:49:31.486 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:49:31.486 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:31.487 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:49:31.488 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.869904_161.1 2021-09-02T07:49:36.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:36.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:36.490 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:49:36.534 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:49:36.534 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:49:36.603 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:49:36.604 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:49:36.604 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:49:36.604 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:49:36.605 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:49:36.605 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:49:36.605 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:36.605 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:36.605 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:49:36.606 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:36.606 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:49:36.606 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:36.606 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:36.607 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:49:36.607 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:36.607 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:49:36.607 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:36.608 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:49:36.608 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:36.608 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:36.608 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:36.609 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:36.609 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:49:36.609 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:36.610 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:36.610 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T07:49:36.610 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:36.610 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:36.611 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:36.611 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T07:49:36.611 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:36.612 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:36.612 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:36.612 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:36.613 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:49:36.613 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:36.613 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:36.614 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:36.614 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:36.614 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:36.615 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:36.615 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:36.616 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:36.616 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:36.617 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:36.617 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:36.618 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:49:36.618 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:36.618 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:36.619 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:36.619 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:36.619 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:36.620 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:36.620 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:36.620 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:36.620 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:36.621 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:36.621 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:49:36.621 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:36.622 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:49:36.622 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:49:36.622 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:49:36.622 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:36.623 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:49:36.624 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568127.103991_175.1 2021-09-02T07:49:41.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:41.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:41.627 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:49:41.720 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:49:41.720 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:49:42.178 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:49:42.179 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:49:42.179 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:49:42.179 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:49:42.179 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:49:42.180 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:49:42.180 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:42.180 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:42.180 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:49:42.181 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:42.181 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:49:42.181 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:42.181 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:42.182 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:49:42.182 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:42.182 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:49:42.182 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:42.182 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:49:42.183 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:42.183 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:42.183 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:42.183 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:42.183 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:49:42.184 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:42.184 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:42.184 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.911309_165.1", 2021-09-02T07:49:42.185 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:42.185 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:42.185 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:42.185 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.911309Z", 2021-09-02T07:49:42.186 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:42.186 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:42.186 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:42.187 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:42.187 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:49:42.187 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:42.187 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:42.187 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:42.188 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:42.188 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:42.188 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:42.189 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:42.189 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:42.189 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:42.189 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:42.190 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:42.190 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:49:42.190 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:42.190 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:42.191 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:42.191 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:42.191 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:42.191 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:42.192 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:42.192 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:42.192 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:42.192 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:42.192 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:49:42.193 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:42.193 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:49:42.193 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:49:42.193 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:49:42.194 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:42.194 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:49:42.196 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.911309_165.1 2021-09-02T07:49:47.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:47.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:47.199 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:49:47.254 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:49:47.255 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:49:47.334 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:49:47.334 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:49:47.334 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:49:47.335 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:49:47.335 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:49:47.335 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:49:47.335 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:47.335 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:47.336 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:49:47.336 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:47.336 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:49:47.336 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:47.337 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:47.337 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:49:47.337 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:47.337 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:49:47.337 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:47.338 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:49:47.338 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:47.338 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:47.338 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:47.339 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:47.339 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:49:47.339 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:47.339 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:47.339 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T07:49:47.345 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:47.346 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:47.346 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:47.346 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T07:49:47.346 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:47.347 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:47.347 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:47.347 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:47.347 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:49:47.347 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:47.348 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:47.348 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:47.348 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:47.348 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:47.349 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:47.349 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:47.349 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:47.350 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:47.350 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:47.350 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:47.351 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:49:47.352 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:47.352 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:47.352 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:47.352 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:47.353 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:47.353 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:47.353 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:47.353 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:47.354 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:47.354 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:47.354 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:49:47.354 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:47.354 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:49:47.355 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:49:47.355 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:49:47.355 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:47.355 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:49:47.357 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568127.103991_175.1 2021-09-02T07:49:52.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:52.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:52.360 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:49:52.436 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:49:52.436 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:49:53.005 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:49:53.005 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:49:53.005 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:49:53.006 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:49:53.006 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:49:53.006 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:49:53.006 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:53.007 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:53.007 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:49:53.007 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:53.007 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:49:53.008 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:53.008 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:53.008 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:49:53.008 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:53.009 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:49:53.009 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:53.009 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:49:53.009 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:53.010 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:53.010 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:53.010 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:53.011 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:49:53.012 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:53.012 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:53.012 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.844214_158.1", 2021-09-02T07:49:53.012 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:53.013 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:53.013 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:53.013 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.844214Z", 2021-09-02T07:49:53.013 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:53.014 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:53.014 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:53.014 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:53.014 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:49:53.015 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:53.015 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:53.015 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:53.015 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:53.015 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:53.016 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:53.016 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:53.016 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:53.016 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:53.017 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:53.017 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:53.017 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:49:53.017 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:53.018 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:53.018 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:53.018 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:53.018 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:53.018 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:53.019 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:53.019 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:53.019 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:53.019 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:53.020 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:49:53.020 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:53.020 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:49:53.020 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:49:53.021 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:49:53.021 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:53.021 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:49:53.022 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.844214_158.1 2021-09-02T07:49:58.023 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:58.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:49:58.024 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:49:58.075 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:49:58.075 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:49:58.349 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:49:58.350 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:49:58.350 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:49:58.350 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:49:58.350 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:49:58.351 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:49:58.351 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:58.351 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:58.351 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:49:58.352 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:58.352 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:49:58.352 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:58.352 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:58.353 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:49:58.353 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:58.353 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:49:58.353 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:58.354 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:49:58.354 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:58.354 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:58.354 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:58.355 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:58.355 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:49:58.355 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:58.355 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:58.355 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T07:49:58.356 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:58.356 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:58.356 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:58.356 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T07:49:58.357 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:58.357 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:58.357 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:58.358 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:58.358 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:49:58.358 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:58.358 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:58.359 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:58.359 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:58.359 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:58.360 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:58.360 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:58.360 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:58.360 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:58.361 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:58.361 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:49:58.361 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:49:58.361 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:49:58.362 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:49:58.362 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:49:58.362 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:49:58.362 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:49:58.363 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:49:58.363 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:49:58.363 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:49:58.363 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:58.364 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:58.364 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:49:58.364 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:49:58.364 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:49:58.365 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:49:58.365 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:49:58.365 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:49:58.365 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:49:58.366 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.906521_163.1 2021-09-02T07:50:03.368 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:03.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:03.370 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:50:03.417 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:50:03.418 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:50:03.491 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:50:03.491 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:50:03.492 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:50:03.492 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:50:03.492 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:50:03.492 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:50:03.493 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:03.493 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:03.493 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:50:03.493 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:03.493 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:50:03.494 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:03.494 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:03.494 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:50:03.494 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:03.495 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:50:03.495 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:03.495 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:50:03.495 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:03.496 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:03.496 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:03.496 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:03.496 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:50:03.496 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:03.497 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:03.497 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.914475_166.1", 2021-09-02T07:50:03.497 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:03.497 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:03.498 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:03.498 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.914475Z", 2021-09-02T07:50:03.498 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:03.498 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:03.499 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:03.499 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:03.499 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:50:03.499 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:03.501 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:03.501 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:03.501 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:03.501 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:03.502 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:03.502 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:03.502 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:03.502 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:03.503 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:03.503 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:03.503 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:50:03.503 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:03.503 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:03.504 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:03.504 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:03.504 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:03.504 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:03.505 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:03.505 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:03.505 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:03.505 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:03.506 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:50:03.506 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:03.506 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:50:03.506 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:50:03.506 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:50:03.507 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:03.507 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:50:03.508 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.914475_166.1 2021-09-02T07:50:08.509 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:08.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:08.511 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:50:08.556 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:50:08.556 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:50:08.628 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:50:08.628 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:50:08.628 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:50:08.628 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:50:08.629 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:50:08.629 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:50:08.629 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:08.629 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:08.630 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:50:08.630 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:08.630 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:50:08.630 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:08.631 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:08.631 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:50:08.631 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:08.632 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:50:08.632 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:08.632 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:50:08.632 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:08.632 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:08.633 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:08.633 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:08.633 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:50:08.633 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:08.634 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:08.634 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T07:50:08.634 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:08.634 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:08.635 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:08.635 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T07:50:08.635 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:08.635 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:08.636 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:08.636 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:08.636 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:50:08.636 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:08.636 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:08.637 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:08.637 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:08.637 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:08.637 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:08.638 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:08.638 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:08.638 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:08.638 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:08.638 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:08.639 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:50:08.639 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:08.639 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:08.639 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:08.640 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:08.640 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:08.640 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:08.640 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:08.640 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:08.641 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:08.641 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:08.641 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:50:08.641 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:08.642 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:50:08.642 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:50:08.642 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:50:08.642 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:08.642 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:50:08.644 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.875533_162.1 2021-09-02T07:50:13.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:13.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:13.647 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:50:13.724 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:50:13.724 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:50:13.931 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:50:13.931 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:50:13.931 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:50:13.932 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:50:13.932 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:50:13.932 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:50:13.932 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:13.932 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:13.933 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:50:13.933 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:13.933 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:50:13.933 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:13.934 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:13.934 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:50:13.934 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:13.934 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:50:13.935 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:13.935 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:50:13.935 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:13.935 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:13.935 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:13.936 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:13.936 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:50:13.936 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:13.936 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:13.937 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.862512_159.1", 2021-09-02T07:50:13.937 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:13.937 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:13.937 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:13.937 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.862512Z", 2021-09-02T07:50:13.938 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:13.938 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:13.938 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:13.939 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:13.939 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:50:13.939 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:13.939 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:13.940 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:13.940 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:13.941 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:13.941 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:13.941 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:13.941 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:13.941 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:13.942 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:13.942 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:13.942 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:50:13.942 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:13.943 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:13.943 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:13.943 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:13.943 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:13.943 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:13.944 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:13.944 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:13.944 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:13.944 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:13.945 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:50:13.945 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:13.945 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:50:13.945 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:50:13.946 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:50:13.946 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:13.946 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:50:13.947 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.862512_159.1 2021-09-02T07:50:18.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:18.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:18.950 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:50:19.004 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:50:19.005 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:50:19.234 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:50:19.234 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:50:19.234 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:50:19.235 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:50:19.235 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:50:19.235 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:50:19.235 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:19.235 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:19.236 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:50:19.236 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:19.236 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:50:19.236 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:19.236 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:19.237 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:50:19.237 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:19.237 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:50:19.237 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:19.237 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:50:19.238 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:19.238 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:19.238 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:19.238 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:19.239 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:50:19.239 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:19.239 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:19.239 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T07:50:19.239 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:19.240 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:19.240 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:19.240 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T07:50:19.240 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:19.241 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:19.241 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:19.241 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:19.241 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:50:19.241 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:19.242 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:19.242 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:19.242 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:19.242 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:19.243 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:19.243 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:19.243 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:19.243 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:19.243 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:19.244 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:19.244 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:50:19.244 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:19.244 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:19.244 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:19.245 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:19.245 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:19.245 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:19.245 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:19.245 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:19.246 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:19.246 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:19.246 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:50:19.246 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:19.247 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:50:19.247 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:50:19.247 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:50:19.247 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:19.247 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:50:19.248 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.910082_164.1 2021-09-02T07:50:24.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:24.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:24.250 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:50:24.341 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:50:24.341 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:50:24.758 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:50:24.758 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:50:24.758 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:50:24.759 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:50:24.759 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:50:24.759 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:50:24.759 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:24.760 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:24.760 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:50:24.760 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:24.760 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:50:24.761 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:24.761 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:24.761 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:50:24.761 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:24.761 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:50:24.762 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:24.762 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:50:24.762 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:24.762 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:24.763 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:24.763 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:24.763 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:50:24.763 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:24.763 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:24.764 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T07:50:24.764 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:24.764 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:24.764 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:24.765 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T07:50:24.765 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:24.765 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:24.765 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:24.766 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:24.766 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:50:24.766 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:24.767 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:24.767 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:24.767 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:24.767 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:24.768 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:24.768 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:24.768 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:24.768 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:24.769 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:24.769 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:24.769 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:50:24.769 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:24.770 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:24.770 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:24.770 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:24.770 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:24.770 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:24.771 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:24.771 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:24.771 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:24.771 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:24.772 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:50:24.772 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:24.772 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:50:24.772 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:50:24.773 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:50:24.773 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:24.773 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:50:24.774 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568127.103991_175.1 2021-09-02T07:50:29.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:29.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:29.776 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:50:29.881 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:50:29.882 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:50:30.272 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:50:30.272 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:50:30.273 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:50:30.273 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:50:30.273 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:50:30.273 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:50:30.274 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:30.274 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:30.274 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:50:30.274 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:30.274 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:50:30.275 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:30.275 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:30.275 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:50:30.275 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:30.276 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:50:30.276 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:30.276 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:50:30.276 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:30.276 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:30.277 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:30.277 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:30.277 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:50:30.277 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:30.278 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:30.278 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.844214_158.1", 2021-09-02T07:50:30.278 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:30.278 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:30.278 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:30.279 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.844214Z", 2021-09-02T07:50:30.279 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:30.279 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:30.279 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:30.280 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:30.280 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:50:30.280 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:30.280 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:30.280 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:30.281 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:30.281 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:30.281 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:30.281 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:30.282 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:30.282 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:30.282 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:30.282 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:30.282 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:50:30.283 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:30.283 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:30.283 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:30.283 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:30.284 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:30.284 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:30.284 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:30.284 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:30.284 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:30.285 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:30.285 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:50:30.285 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:30.285 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:50:30.286 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:50:30.286 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:50:30.286 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:30.286 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:50:30.288 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.844214_158.1 2021-09-02T07:50:35.289 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:35.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:35.291 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:50:35.341 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:50:35.342 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:50:35.418 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:50:35.419 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:50:35.419 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:50:35.419 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:50:35.419 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:50:35.420 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:50:35.420 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:35.420 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:35.420 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:50:35.420 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:35.421 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:50:35.421 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:35.421 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:35.421 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:50:35.421 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:35.422 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:50:35.422 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:35.422 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:50:35.422 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:35.423 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:35.423 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:35.423 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:35.423 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:50:35.424 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:35.424 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:35.424 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.914475_166.1", 2021-09-02T07:50:35.424 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:35.424 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:35.425 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:35.425 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.914475Z", 2021-09-02T07:50:35.425 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:35.425 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:35.425 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:35.426 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:35.426 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:50:35.426 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:35.426 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:35.427 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:35.427 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:35.427 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:35.427 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:35.427 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:35.428 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:35.428 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:35.428 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:35.428 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:35.429 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:50:35.429 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:35.429 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:35.429 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:35.429 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:35.430 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:35.430 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:35.430 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:35.430 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:35.431 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:35.431 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:35.431 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:50:35.431 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:35.432 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:50:35.432 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:50:35.432 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:50:35.432 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:35.433 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:50:35.434 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.914475_166.1 2021-09-02T07:50:40.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:40.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:40.437 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:50:40.485 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:50:40.485 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:50:40.569 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:50:40.569 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:50:40.569 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:50:40.569 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:50:40.570 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:50:40.570 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:50:40.570 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:40.570 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:40.571 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:50:40.571 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:40.571 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:50:40.571 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:40.572 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:40.572 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:50:40.572 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:40.572 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:50:40.573 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:40.573 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:50:40.573 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:40.573 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:40.574 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:40.574 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:40.574 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:50:40.574 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:40.574 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:40.575 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.844214_158.1", 2021-09-02T07:50:40.575 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:40.575 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:40.575 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:40.576 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.844214Z", 2021-09-02T07:50:40.576 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:40.576 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:40.576 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:40.577 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:40.577 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:50:40.577 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:40.577 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:40.578 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:40.578 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:40.578 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:40.578 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:40.579 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:40.579 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:40.579 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:40.579 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:40.580 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:40.580 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:50:40.580 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:40.580 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:40.581 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:40.581 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:40.581 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:40.581 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:40.581 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:40.582 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:40.582 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:40.582 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:40.582 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:50:40.583 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:40.583 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:50:40.583 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:50:40.583 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:50:40.584 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:40.584 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:50:40.585 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.844214_158.1 2021-09-02T07:50:45.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:45.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:45.588 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:50:45.633 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:50:45.634 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:50:45.696 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:50:45.698 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:50:45.698 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:50:45.699 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:50:45.700 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:50:45.700 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:50:45.700 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:45.701 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:45.701 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:50:45.701 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:45.702 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:50:45.702 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:45.703 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:45.703 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:50:45.703 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:45.703 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:50:45.703 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:45.704 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:50:45.704 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:45.704 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:45.704 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:45.705 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:45.705 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:50:45.705 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:45.706 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:45.706 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.985199_169.1", 2021-09-02T07:50:45.706 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:45.706 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:45.707 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:45.707 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.985199Z", 2021-09-02T07:50:45.707 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:45.707 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:45.708 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:45.708 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:45.708 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:50:45.709 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:45.709 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:45.709 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:45.709 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:45.710 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:45.710 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:45.710 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:45.710 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:45.711 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:45.711 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:45.711 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:45.711 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:50:45.711 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:45.712 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:45.712 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:45.712 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:45.713 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:45.713 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:45.713 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:45.714 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:45.714 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:45.714 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:45.714 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:50:45.715 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:45.715 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:50:45.715 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:50:45.715 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:50:45.716 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:45.716 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:50:45.717 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.985199_169.1 2021-09-02T07:50:50.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:50.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:50.721 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:50:50.780 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:50:50.781 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:50:50.885 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:50:50.886 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:50:50.886 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:50:50.886 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:50:50.887 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:50:50.887 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:50:50.887 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:50.887 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:50.888 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:50:50.888 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:50.888 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:50:50.888 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:50.889 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:50.889 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:50:50.889 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:50.889 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:50:50.889 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:50.890 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:50:50.890 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:50.890 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:50.890 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:50.891 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:50.891 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:50:50.891 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:50.891 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:50.892 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T07:50:50.892 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:50.892 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:50.892 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:50.892 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T07:50:50.893 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:50.893 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:50.893 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:50.893 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:50.894 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:50:50.894 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:50.894 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:50.895 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:50.895 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:50.895 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:50.895 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:50.896 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:50.896 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:50.896 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:50.896 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:50.897 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:50.897 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:50:50.897 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:50.897 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:50.897 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:50.898 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:50.898 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:50.898 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:50.899 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:50.899 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:50.899 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:50.899 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:50.900 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:50:50.900 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:50.900 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:50:50.900 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:50:50.900 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:50:50.901 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:50.901 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:50:50.902 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.910082_164.1 2021-09-02T07:50:55.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:55.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:50:55.906 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:50:55.996 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:50:55.996 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:50:56.464 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:50:56.465 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:50:56.465 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:50:56.465 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:50:56.466 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:50:56.466 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:50:56.467 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:56.467 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:56.467 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:50:56.467 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:56.467 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:50:56.468 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:56.468 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:56.468 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:50:56.469 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:56.469 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:50:56.469 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:56.469 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:50:56.470 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:56.470 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:56.470 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:56.470 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:56.471 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:50:56.471 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:56.471 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:56.471 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T07:50:56.471 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:56.472 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:56.472 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:56.472 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T07:50:56.472 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:56.473 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:56.473 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:56.473 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:56.473 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:50:56.474 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:56.474 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:56.474 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:56.474 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:56.475 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:56.475 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:56.475 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:56.476 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:56.476 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:56.476 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:56.476 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:50:56.476 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:50:56.477 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:50:56.477 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:50:56.477 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:50:56.477 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:50:56.478 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:50:56.478 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:50:56.478 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:50:56.478 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:50:56.479 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:56.479 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:56.479 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:50:56.479 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:50:56.480 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:50:56.480 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:50:56.480 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:50:56.480 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:50:56.481 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:50:56.482 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.875533_162.1 2021-09-02T07:51:01.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:01.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:01.485 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:51:01.571 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:51:01.571 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:51:02.169 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:51:02.169 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:51:02.170 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:51:02.170 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:51:02.170 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:51:02.170 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:51:02.171 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:02.171 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:02.171 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:51:02.171 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:02.171 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:51:02.172 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:02.172 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:02.172 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:51:02.172 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:02.172 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:51:02.173 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:02.173 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:51:02.173 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:02.173 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:02.173 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:02.174 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:02.174 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:51:02.174 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:02.174 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:02.175 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.862512_159.1", 2021-09-02T07:51:02.175 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:02.175 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:02.175 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:02.175 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.862512Z", 2021-09-02T07:51:02.176 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:02.176 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:02.176 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:02.176 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:02.176 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:51:02.177 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:02.177 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:02.177 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:02.177 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:02.177 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:02.178 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:02.178 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:02.178 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:02.178 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:02.179 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:02.179 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:02.179 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:51:02.179 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:02.179 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:02.180 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:02.180 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:02.180 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:02.180 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:02.181 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:02.181 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:02.181 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:02.181 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:02.181 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:51:02.182 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:02.182 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:51:02.182 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:51:02.182 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:51:02.182 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:02.183 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:51:02.184 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.862512_159.1 2021-09-02T07:51:07.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:07.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:07.187 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:51:07.284 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:51:07.284 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:51:07.701 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:51:07.701 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:51:07.701 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:51:07.702 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:51:07.702 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:51:07.702 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:51:07.702 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:07.703 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:07.703 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:51:07.703 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:07.703 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:51:07.704 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:07.704 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:07.704 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:51:07.704 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:07.705 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:51:07.705 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:07.705 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:51:07.705 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:07.706 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:07.706 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:07.706 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:07.706 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:51:07.707 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:07.707 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:07.707 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.844214_158.1", 2021-09-02T07:51:07.707 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:07.708 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:07.708 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:07.708 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.844214Z", 2021-09-02T07:51:07.708 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:07.709 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:07.709 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:07.709 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:07.709 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:51:07.710 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:07.710 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:07.710 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:07.710 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:07.710 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:07.711 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:07.711 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:07.711 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:07.711 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:07.712 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:07.712 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:07.712 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:51:07.712 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:07.713 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:07.713 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:07.713 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:07.713 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:07.714 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:07.714 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:07.714 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:07.714 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:07.715 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:07.715 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:51:07.715 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:07.715 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:51:07.716 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:51:07.716 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:51:07.716 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:07.716 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:51:07.718 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.844214_158.1 2021-09-02T07:51:12.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:12.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:12.721 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:51:12.818 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:51:12.818 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:51:13.253 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:51:13.253 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:51:13.253 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:51:13.254 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:51:13.254 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:51:13.254 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:51:13.254 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:13.255 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:13.255 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:51:13.255 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:13.255 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:51:13.256 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:13.256 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:13.256 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:51:13.256 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:13.256 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:51:13.257 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:13.257 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:51:13.257 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:13.257 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:13.258 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:13.258 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:13.258 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:51:13.258 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:13.258 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:13.259 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.124227_177.1", 2021-09-02T07:51:13.259 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:13.259 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:13.259 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:13.260 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.124227Z", 2021-09-02T07:51:13.260 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:13.260 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:13.260 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:13.260 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:13.261 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:51:13.261 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:13.261 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:13.262 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:13.262 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:13.262 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:13.262 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:13.262 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:13.263 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:13.263 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:13.263 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:13.263 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:13.264 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:51:13.264 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:13.264 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:13.264 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:13.264 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:13.265 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:13.265 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:13.265 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:13.265 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:13.265 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:13.266 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:13.266 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:51:13.266 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:13.266 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:51:13.267 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:51:13.267 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:51:13.267 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:13.267 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:51:13.269 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568127.124227_177.1 2021-09-02T07:51:18.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:18.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:18.272 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:51:18.340 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:51:18.341 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:51:19.003 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:51:19.112 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:51:19.113 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:51:19.114 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:51:19.114 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:51:19.115 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:51:19.115 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:19.116 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:19.118 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:51:19.119 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:19.119 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:51:19.119 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:19.120 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:19.120 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:51:19.121 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:19.121 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:51:19.121 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:19.123 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:51:19.123 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:19.123 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:19.124 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:19.125 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:19.125 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:51:19.126 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:19.126 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:19.126 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.869904_161.1", 2021-09-02T07:51:19.127 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:19.128 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:19.128 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:19.128 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.869904Z", 2021-09-02T07:51:19.129 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:19.129 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:19.129 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:19.129 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:19.130 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:51:19.130 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:19.130 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:19.131 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:19.131 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:19.131 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:19.131 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:19.131 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:19.132 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:19.132 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:19.132 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:19.132 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:19.133 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:51:19.133 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:19.133 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:19.133 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:19.133 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:19.134 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:19.134 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:19.134 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:19.134 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:19.135 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:19.135 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:19.135 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:51:19.135 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:19.135 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:51:19.136 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:51:19.136 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:51:19.136 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:19.136 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:51:19.138 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.869904_161.1 2021-09-02T07:51:24.139 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:24.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:24.140 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:51:24.239 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:51:24.240 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:51:24.708 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:51:24.709 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:51:24.709 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:51:24.709 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:51:24.710 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:51:24.710 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:51:24.710 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:24.710 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:24.711 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:51:24.711 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:24.711 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:51:24.711 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:24.712 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:24.712 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:51:24.712 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:24.712 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:51:24.713 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:24.713 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:51:24.713 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:24.713 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:24.713 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:24.714 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:24.714 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:51:24.714 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:24.714 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:24.715 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.249624_187.1", 2021-09-02T07:51:24.715 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:24.715 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:24.715 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:24.716 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.249624Z", 2021-09-02T07:51:24.716 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:24.716 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:24.716 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:24.717 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:24.717 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:51:24.717 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:24.717 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:24.717 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:24.718 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:24.718 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:24.718 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:24.718 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:24.719 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:24.719 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:24.719 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:24.720 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:24.720 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:51:24.720 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:24.720 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:24.720 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:24.721 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:24.721 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:24.721 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:24.721 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:24.722 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:24.722 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:24.722 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:24.722 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:51:24.723 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:24.723 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:51:24.723 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:51:24.723 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:51:24.724 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:24.724 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:51:24.725 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568127.249624_187.1 2021-09-02T07:51:29.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:29.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:29.728 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:51:29.773 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:51:29.773 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:51:29.949 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:51:29.949 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:51:29.950 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:51:29.950 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:51:29.950 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:51:29.951 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:51:29.951 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:29.951 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:29.951 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:51:29.952 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:29.952 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:51:29.952 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:29.952 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:29.953 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:51:29.953 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:29.953 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:51:29.953 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:29.954 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:51:29.954 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:29.954 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:29.954 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:29.954 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:29.955 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:51:29.955 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:29.955 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:29.955 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T07:51:29.956 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:29.956 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:29.956 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:29.956 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T07:51:29.957 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:29.957 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:29.957 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:29.957 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:29.957 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:51:29.958 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:29.958 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:29.958 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:29.958 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:29.959 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:29.959 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:29.959 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:29.959 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:29.960 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:29.960 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:29.960 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:29.960 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:51:29.961 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:29.961 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:29.961 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:29.961 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:29.962 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:29.962 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:29.962 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:29.962 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:29.963 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:29.963 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:29.963 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:51:29.963 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:29.963 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:51:29.964 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:51:29.964 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:51:29.964 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:29.964 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:51:29.965 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.875533_162.1 2021-09-02T07:51:34.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:34.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:34.968 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:51:35.018 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:51:35.018 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:51:35.091 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:51:35.091 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:51:35.091 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:51:35.092 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:51:35.092 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:51:35.092 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:51:35.092 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:35.093 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:35.093 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:51:35.093 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:35.093 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:51:35.093 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:35.094 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:35.094 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:51:35.094 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:35.094 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:51:35.095 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:35.095 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:51:35.095 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:35.095 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:35.095 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:35.096 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:35.096 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:51:35.096 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:35.096 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:35.097 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T07:51:35.097 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:35.097 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:35.097 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:35.097 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T07:51:35.098 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:35.098 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:35.098 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:35.098 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:35.099 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:51:35.099 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:35.099 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:35.099 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:35.099 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:35.100 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:35.100 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:35.100 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:35.100 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:35.101 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:35.101 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:35.101 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:35.101 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:51:35.101 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:35.102 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:35.102 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:35.102 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:35.102 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:35.103 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:35.103 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:35.103 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:35.103 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:35.104 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:35.104 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:51:35.104 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:35.104 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:51:35.104 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:51:35.105 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:51:35.105 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:35.105 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:51:35.106 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.910082_164.1 2021-09-02T07:51:40.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:40.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:40.109 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:51:40.153 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:51:40.153 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:51:40.233 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:51:40.233 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:51:40.233 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:51:40.233 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:51:40.234 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:51:40.234 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:51:40.234 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:40.235 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:40.241 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:51:40.242 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:40.242 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:51:40.242 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:40.242 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:40.243 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:51:40.243 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:40.243 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:51:40.243 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:40.243 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:51:40.244 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:40.244 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:40.244 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:40.244 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:40.245 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:51:40.245 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:40.245 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:40.245 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T07:51:40.245 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:40.246 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:40.246 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:40.246 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T07:51:40.246 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:40.247 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:40.247 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:40.247 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:40.247 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:51:40.247 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:40.248 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:40.248 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:40.248 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:40.248 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:40.249 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:40.249 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:40.249 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:40.249 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:40.249 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:40.250 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:40.250 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:51:40.250 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:40.250 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:40.251 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:40.251 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:40.251 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:40.251 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:40.251 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:40.252 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:40.252 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:40.252 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:40.252 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:51:40.253 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:40.253 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:51:40.253 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:51:40.253 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:51:40.253 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:40.254 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:51:40.255 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.906521_163.1 2021-09-02T07:51:45.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:45.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:45.257 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:51:45.340 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:51:45.340 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:51:45.666 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:51:45.667 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:51:45.667 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:51:45.668 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:51:45.668 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:51:45.668 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:51:45.669 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:45.669 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:45.670 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:51:45.670 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:45.671 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:51:45.671 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:45.672 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:45.672 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:51:45.672 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:45.673 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:51:45.673 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:45.674 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:51:45.674 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:45.675 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:45.675 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:45.676 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:45.676 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:51:45.676 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:45.677 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:45.677 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T07:51:45.678 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:45.678 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:45.679 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:45.679 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T07:51:45.680 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:45.680 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:45.681 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:45.682 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:45.682 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:51:45.682 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:45.683 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:45.683 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:45.684 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:45.684 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:45.685 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:45.685 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:45.685 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:45.686 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:45.687 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:45.687 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:45.688 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:51:45.688 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:45.689 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:45.689 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:45.689 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:45.690 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:45.691 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:45.691 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:45.691 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:45.692 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:45.692 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:45.693 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:51:45.693 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:45.694 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:51:45.694 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:51:45.696 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:51:45.696 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:45.697 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:51:45.697 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.875533_162.1 2021-09-02T07:51:50.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:50.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:50.700 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:51:50.758 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:51:50.758 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:51:51.092 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:51:51.092 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:51:51.093 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:51:51.093 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:51:51.093 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:51:51.093 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:51:51.094 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:51.094 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:51.094 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:51:51.094 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:51.095 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:51:51.095 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:51.095 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:51.095 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:51:51.095 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:51.096 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:51:51.096 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:51.096 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:51:51.096 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:51.097 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:51.097 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:51.097 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:51.097 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:51:51.097 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:51.098 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:51.098 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.391532_193.1", 2021-09-02T07:51:51.098 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:51.098 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:51.099 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:51.099 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.391532Z", 2021-09-02T07:51:51.099 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:51.099 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:51.099 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:51.100 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:51.100 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:51:51.100 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:51.100 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:51.100 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:51.101 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:51.101 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:51.101 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:51.101 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:51.102 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:51.102 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:51.102 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:51.102 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:51.102 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:51:51.103 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:51.103 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:51.103 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:51.103 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:51.104 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:51.104 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:51.104 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:51.104 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:51.105 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:51.105 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:51.105 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:51:51.105 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:51.105 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:51:51.106 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:51:51.106 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:51:51.106 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:51.106 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:51:51.107 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568127.391532_193.1 2021-09-02T07:51:56.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:56.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:51:56.110 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:51:56.206 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:51:56.207 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:51:56.452 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:51:56.452 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:51:56.452 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:51:56.453 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:51:56.453 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:51:56.453 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:51:56.453 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:56.453 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:56.454 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:51:56.454 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:56.454 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:51:56.454 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:56.455 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:56.455 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:51:56.455 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:56.455 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:51:56.455 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:56.456 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:51:56.456 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:56.456 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:56.456 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:56.457 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:56.457 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:51:56.457 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:56.457 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:56.458 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.844214_158.1", 2021-09-02T07:51:56.458 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:56.458 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:56.458 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:56.458 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.844214Z", 2021-09-02T07:51:56.459 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:56.459 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:56.459 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:56.459 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:56.460 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:51:56.460 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:56.460 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:56.461 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:56.461 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:56.461 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:56.461 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:56.461 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:56.462 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:56.462 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:56.462 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:56.462 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:51:56.463 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:51:56.463 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:51:56.463 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:51:56.463 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:51:56.464 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:51:56.464 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:51:56.464 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:51:56.464 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:51:56.464 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:51:56.465 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:56.465 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:56.465 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:51:56.465 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:51:56.466 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:51:56.466 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:51:56.466 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:51:56.466 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:51:56.466 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:51:56.468 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.844214_158.1 2021-09-02T07:52:01.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:01.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:01.471 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:52:01.548 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:52:01.548 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:52:01.975 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:52:01.975 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:52:01.976 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:52:01.976 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:52:01.976 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:52:01.976 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:52:01.977 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:01.977 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:01.978 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:52:01.978 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:01.978 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:52:01.978 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:01.979 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:01.979 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:52:01.979 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:01.979 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:52:01.979 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:01.980 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:52:01.980 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:01.980 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:01.980 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:01.981 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:01.981 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:52:01.981 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:01.981 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:01.981 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T07:52:01.982 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:01.982 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:01.982 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:01.982 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T07:52:01.983 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:01.983 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:01.983 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:01.983 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:01.983 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:52:01.984 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:01.984 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:01.984 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:01.984 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:01.985 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:01.985 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:01.985 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:01.985 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:01.986 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:01.986 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:01.986 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:01.986 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:52:01.987 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:01.987 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:01.987 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:01.987 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:01.988 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:01.988 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:01.988 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:01.988 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:01.989 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:01.989 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:01.989 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:52:01.989 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:01.990 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:52:01.990 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:52:01.990 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:52:01.990 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:01.990 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:52:01.992 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568127.103991_175.1 2021-09-02T07:52:06.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:06.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:06.994 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:52:07.054 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:52:07.055 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:52:07.532 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:52:07.533 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:52:07.533 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:52:07.533 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:52:07.534 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:52:07.534 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:52:07.534 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:07.534 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:07.534 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:52:07.535 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:07.535 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:52:07.535 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:07.535 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:07.536 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:52:07.536 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:07.536 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:52:07.536 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:07.536 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:52:07.537 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:07.537 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:07.537 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:07.537 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:07.538 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:52:07.538 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:07.538 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:07.538 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.869904_161.1", 2021-09-02T07:52:07.538 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:07.539 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:07.539 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:07.539 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.869904Z", 2021-09-02T07:52:07.539 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:07.540 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:07.540 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:07.540 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:07.540 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:52:07.540 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:07.541 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:07.541 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:07.541 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:07.541 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:07.542 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:07.542 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:07.542 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:07.542 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:07.542 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:07.543 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:07.543 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:52:07.543 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:07.543 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:07.544 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:07.544 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:07.544 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:07.544 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:07.544 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:07.545 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:07.545 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:07.545 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:07.545 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:52:07.546 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:07.546 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:52:07.546 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:52:07.546 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:52:07.546 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:07.547 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:52:07.548 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.869904_161.1 2021-09-02T07:52:12.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:12.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:12.552 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:52:12.602 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:52:12.602 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:52:12.682 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:52:12.682 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:52:12.683 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:52:12.683 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:52:12.683 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:52:12.683 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:52:12.684 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:12.684 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:12.684 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:52:12.684 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:12.685 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:52:12.685 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:12.685 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:12.685 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:52:12.686 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:12.686 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:52:12.686 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:12.686 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:52:12.687 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:12.687 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:12.687 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:12.687 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:12.688 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:52:12.688 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:12.688 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:12.688 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T07:52:12.688 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:12.689 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:12.689 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:12.689 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T07:52:12.689 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:12.690 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:12.690 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:12.690 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:12.690 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:52:12.691 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:12.691 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:12.691 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:12.691 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:12.691 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:12.692 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:12.692 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:12.692 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:12.692 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:12.693 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:12.693 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:12.693 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:52:12.693 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:12.694 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:12.694 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:12.694 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:12.694 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:12.695 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:12.695 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:12.695 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:12.695 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:12.696 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:12.696 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:52:12.696 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:12.696 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:52:12.697 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:52:12.697 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:52:12.697 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:12.697 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:52:12.699 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.875533_162.1 2021-09-02T07:52:17.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:17.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:17.702 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:52:17.750 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:52:17.750 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:52:17.829 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:52:17.829 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:52:17.829 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:52:17.830 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:52:17.830 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:52:17.830 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:52:17.830 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:17.831 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:17.831 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:52:17.831 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:17.833 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:52:17.833 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:17.834 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:17.834 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:52:17.834 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:17.834 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:52:17.835 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:17.835 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:52:17.835 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:17.835 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:17.836 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:17.836 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:17.836 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:52:17.836 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:17.837 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:17.837 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.911309_165.1", 2021-09-02T07:52:17.837 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:17.837 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:17.838 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:17.838 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.911309Z", 2021-09-02T07:52:17.838 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:17.838 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:17.838 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:17.839 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:17.839 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:52:17.839 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:17.839 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:17.840 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:17.840 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:17.840 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:17.841 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:17.841 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:17.841 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:17.841 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:17.841 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:17.842 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:17.842 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:52:17.843 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:17.843 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:17.843 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:17.844 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:17.844 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:17.844 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:17.844 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:17.845 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:17.845 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:17.845 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:17.845 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:52:17.846 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:17.846 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:52:17.846 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:52:17.846 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:52:17.847 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:17.847 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:52:17.848 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.911309_165.1 2021-09-02T07:52:22.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:22.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:22.852 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:52:22.931 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:52:22.932 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:52:23.273 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:52:23.273 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:52:23.273 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:52:23.273 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:52:23.274 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:52:23.274 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:52:23.274 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:23.275 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:23.275 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:52:23.275 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:23.275 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:52:23.275 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:23.276 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:23.276 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:52:23.276 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:23.276 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:52:23.277 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:23.277 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:52:23.277 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:23.277 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:23.278 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:23.278 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:23.278 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:52:23.278 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:23.279 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:23.279 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.862512_159.1", 2021-09-02T07:52:23.279 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:23.279 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:23.280 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:23.280 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.862512Z", 2021-09-02T07:52:23.280 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:23.280 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:23.281 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:23.281 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:23.281 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:52:23.281 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:23.281 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:23.282 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:23.282 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:23.282 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:23.282 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:23.283 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:23.283 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:23.283 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:23.283 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:23.284 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:23.284 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:52:23.284 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:23.284 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:23.285 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:23.285 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:23.285 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:23.286 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:23.286 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:23.286 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:23.286 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:23.287 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:23.287 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:52:23.287 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:23.287 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:52:23.288 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:52:23.288 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:52:23.288 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:23.288 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:52:23.290 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.862512_159.1 2021-09-02T07:52:28.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:28.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:28.293 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:52:28.358 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:52:28.358 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:52:28.498 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:52:28.499 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:52:28.499 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:52:28.499 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:52:28.500 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:52:28.500 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:52:28.500 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:28.500 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:28.500 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:52:28.501 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:28.501 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:52:28.501 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:28.502 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:28.502 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:52:28.502 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:28.502 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:52:28.503 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:28.503 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:52:28.503 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:28.503 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:28.504 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:28.504 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:28.504 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:52:28.504 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:28.505 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:28.505 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.844214_158.1", 2021-09-02T07:52:28.505 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:28.506 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:28.506 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:28.506 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.844214Z", 2021-09-02T07:52:28.507 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:28.507 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:28.507 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:28.508 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:28.508 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:52:28.508 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:28.508 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:28.508 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:28.509 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:28.509 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:28.509 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:28.509 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:28.510 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:28.510 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:28.510 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:28.510 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:28.510 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:52:28.511 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:28.511 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:28.511 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:28.511 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:28.512 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:28.512 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:28.512 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:28.513 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:28.513 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:28.513 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:28.513 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:52:28.513 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:28.514 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:52:28.514 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:52:28.514 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:52:28.514 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:28.515 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:52:28.516 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.844214_158.1 2021-09-02T07:52:33.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:33.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:33.519 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:52:33.619 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:52:33.620 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:52:34.010 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:52:34.011 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:52:34.011 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:52:34.011 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:52:34.011 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:52:34.012 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:52:34.012 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:34.012 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:34.012 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:52:34.013 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:34.013 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:52:34.013 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:34.013 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:34.013 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:52:34.014 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:34.014 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:52:34.014 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:34.014 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:52:34.015 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:34.015 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:34.015 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:34.015 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:34.016 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:52:34.016 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:34.016 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:34.016 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.129118_179.1", 2021-09-02T07:52:34.016 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:34.017 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:34.017 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:34.017 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.129118Z", 2021-09-02T07:52:34.017 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:34.018 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:34.018 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:34.018 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:34.018 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:52:34.019 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:34.019 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:34.019 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:34.019 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:34.019 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:34.020 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:34.020 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:34.020 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:34.020 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:34.021 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:34.021 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:34.021 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:52:34.021 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:34.021 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:34.022 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:34.022 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:34.022 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:34.022 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:34.023 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:34.023 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:34.023 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:34.023 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:34.023 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:52:34.024 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:34.024 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:52:34.024 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:52:34.024 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:52:34.025 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:34.025 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:52:34.026 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568127.129118_179.1 2021-09-02T07:52:39.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:39.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:39.029 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:52:39.092 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:52:39.092 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:52:39.801 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:52:39.802 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:52:39.802 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:52:39.802 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:52:39.803 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:52:39.803 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:52:39.803 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:39.804 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:39.804 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:52:39.804 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:39.804 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:52:39.804 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:39.805 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:39.805 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:52:39.805 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:39.805 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:52:39.806 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:39.806 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:52:39.806 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:39.806 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:39.807 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:39.807 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:39.807 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:52:39.807 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:39.807 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:39.808 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.990121_170.1", 2021-09-02T07:52:39.808 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:39.808 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:39.809 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:39.809 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.990121Z", 2021-09-02T07:52:39.809 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:39.810 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:39.810 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:39.810 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:39.810 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:52:39.811 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:39.811 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:39.811 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:39.811 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:39.812 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:39.812 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:39.812 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:39.813 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:39.814 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:39.814 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:39.814 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:39.814 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:52:39.815 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:39.815 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:39.815 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:39.815 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:39.816 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:39.816 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:39.816 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:39.816 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:39.816 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:39.817 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:39.817 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:52:39.817 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:39.818 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:52:39.818 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:52:39.818 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:52:39.819 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:39.819 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:52:39.820 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.990121_170.1 2021-09-02T07:52:44.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:44.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:44.822 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:52:44.900 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:52:44.900 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:52:45.512 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:52:45.512 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:52:45.512 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:52:45.512 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:52:45.513 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:52:45.513 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:52:45.513 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:45.513 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:45.514 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:52:45.514 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:45.514 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:52:45.514 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:45.515 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:45.515 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:52:45.515 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:45.515 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:52:45.515 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:45.516 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:52:45.516 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:45.516 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:45.516 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:45.517 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:45.517 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:52:45.517 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:45.517 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:45.517 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.911309_165.1", 2021-09-02T07:52:45.518 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:45.518 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:45.518 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:45.518 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.911309Z", 2021-09-02T07:52:45.519 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:45.519 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:45.519 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:45.519 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:45.519 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:52:45.520 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:45.520 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:45.520 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:45.520 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:45.521 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:45.521 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:45.521 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:45.521 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:45.521 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:45.522 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:45.522 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:45.522 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:52:45.522 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:45.522 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:45.523 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:45.523 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:45.523 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:45.523 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:45.524 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:45.524 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:45.524 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:45.524 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:45.524 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:52:45.525 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:45.525 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:52:45.525 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:52:45.525 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:52:45.525 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:45.526 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:52:45.527 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.911309_165.1 2021-09-02T07:52:50.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:50.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:50.531 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:52:50.610 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:52:50.611 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:52:51.153 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:52:51.153 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:52:51.153 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:52:51.154 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:52:51.154 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:52:51.154 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:52:51.154 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:51.155 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:51.155 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:52:51.155 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:51.155 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:52:51.156 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:51.156 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:51.156 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:52:51.156 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:51.156 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:52:51.157 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:51.157 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:52:51.157 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:51.157 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:51.158 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:51.158 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:51.158 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:52:51.158 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:51.158 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:51.159 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.862512_159.1", 2021-09-02T07:52:51.159 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:51.159 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:51.159 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:51.160 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.862512Z", 2021-09-02T07:52:51.160 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:51.160 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:51.161 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:51.161 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:51.161 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:52:51.161 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:51.162 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:51.162 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:51.162 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:51.162 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:51.162 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:51.163 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:51.163 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:51.163 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:51.163 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:51.164 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:51.164 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:52:51.164 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:51.164 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:51.164 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:51.165 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:51.165 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:51.165 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:51.165 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:51.166 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:51.166 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:51.166 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:51.166 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:52:51.166 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:51.167 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:52:51.167 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:52:51.167 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:52:51.167 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:51.168 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:52:51.168 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.862512_159.1 2021-09-02T07:52:56.169 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:56.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:52:56.176 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:52:56.225 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:52:56.226 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:52:56.653 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:52:56.654 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:52:56.654 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:52:56.655 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:52:56.655 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:52:56.655 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:52:56.656 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:56.656 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:56.656 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:52:56.657 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:56.657 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:52:56.658 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:56.658 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:56.658 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:52:56.659 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:56.661 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:52:56.662 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:56.662 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:52:56.663 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:56.663 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:56.664 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:56.664 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:56.664 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:52:56.665 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:56.665 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:56.666 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.985199_169.1", 2021-09-02T07:52:56.667 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:56.667 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:56.668 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:56.668 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.985199Z", 2021-09-02T07:52:56.668 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:56.668 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:56.669 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:56.669 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:56.669 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:52:56.669 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:56.669 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:56.670 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:56.670 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:56.670 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:56.670 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:56.671 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:56.671 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:56.671 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:56.671 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:56.671 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:52:56.672 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:52:56.672 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:52:56.672 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:52:56.672 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:52:56.673 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:52:56.673 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:52:56.673 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:52:56.673 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:52:56.673 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:52:56.674 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:56.674 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:56.674 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:52:56.674 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:52:56.674 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:52:56.675 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:52:56.675 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:52:56.675 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:52:56.675 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:52:56.676 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.985199_169.1 2021-09-02T07:53:01.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:53:01.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:53:01.679 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:53:01.727 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:53:01.728 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:53:01.798 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:53:01.798 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:53:01.799 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:53:01.799 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:53:01.799 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:53:01.799 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:53:01.800 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:01.800 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:01.800 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:53:01.800 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:01.801 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:53:01.801 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:01.801 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:01.801 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:53:01.801 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:01.802 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:53:01.802 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:01.802 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:53:01.803 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:01.803 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:01.803 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:01.803 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:01.804 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:53:01.804 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:01.804 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:01.805 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.129118_179.1", 2021-09-02T07:53:01.805 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:01.805 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:01.806 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:01.806 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.129118Z", 2021-09-02T07:53:01.806 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:01.806 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:01.807 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:01.807 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:01.807 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:53:01.807 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:01.807 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:01.808 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:53:01.808 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:01.808 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:01.808 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:01.809 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:53:01.809 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:01.809 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:01.809 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:01.809 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:01.810 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:53:01.810 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:01.810 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:01.810 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:53:01.811 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:01.811 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:01.811 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:01.811 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:53:01.811 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:01.812 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:01.812 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:01.812 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:53:01.812 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:01.813 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:53:01.813 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:53:01.813 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:53:01.813 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:01.813 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:53:01.815 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568127.129118_179.1 2021-09-02T07:53:06.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:53:06.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:53:06.818 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:53:06.861 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:53:06.861 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:53:06.932 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:53:06.933 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:53:06.933 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:53:06.933 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:53:06.933 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:53:06.934 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:53:06.934 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:06.934 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:06.934 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:53:06.935 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:06.935 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:53:06.935 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:06.935 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:06.936 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:53:06.936 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:06.936 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:53:06.936 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:06.936 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:53:06.937 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:06.937 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:06.937 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:06.937 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:06.938 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:53:06.938 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:06.942 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:06.943 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T07:53:06.943 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:06.943 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:06.943 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:06.944 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T07:53:06.944 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:06.944 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:06.944 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:06.944 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:06.945 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:53:06.945 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:06.946 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:06.946 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:53:06.946 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:06.946 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:06.946 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:06.947 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:53:06.947 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:06.947 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:06.948 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:06.948 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:06.948 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:53:06.948 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:06.949 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:06.949 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:53:06.949 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:06.950 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:06.950 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:06.950 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:53:06.950 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:06.951 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:06.951 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:06.951 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:53:06.951 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:06.951 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:53:06.952 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:53:06.952 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:53:06.952 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:06.952 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:53:06.954 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.910082_164.1 2021-09-02T07:53:11.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:53:11.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:53:11.956 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:53:12.041 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:53:12.041 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:53:12.288 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:53:12.289 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:53:12.294 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:53:12.295 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:53:12.296 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:53:12.298 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:53:12.299 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:12.300 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:12.300 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:53:12.300 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:12.301 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:53:12.302 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:12.303 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:12.304 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:53:12.304 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:12.304 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:53:12.304 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:12.305 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:53:12.305 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:12.305 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:12.305 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:12.305 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:12.306 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:53:12.306 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:12.306 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:12.306 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.844214_158.1", 2021-09-02T07:53:12.307 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:12.307 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:12.307 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:12.307 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.844214Z", 2021-09-02T07:53:12.307 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:12.308 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:12.308 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:12.308 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:12.308 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:53:12.309 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:12.309 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:12.309 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:53:12.309 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:12.309 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:12.310 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:12.310 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:53:12.310 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:12.310 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:12.311 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:12.311 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:12.311 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:53:12.311 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:12.311 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:12.312 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:53:12.312 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:12.312 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:12.312 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:12.313 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:53:12.313 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:12.313 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:12.313 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:12.313 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:53:12.314 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:12.314 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:53:12.314 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:53:12.314 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:53:12.315 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:12.315 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:53:12.316 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.844214_158.1 2021-09-02T07:53:17.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:53:17.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:53:17.319 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:53:17.379 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:53:17.380 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:53:17.489 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:53:17.489 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:53:17.490 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:53:17.490 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:53:17.490 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:53:17.490 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:53:17.491 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:17.491 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:17.491 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:53:17.491 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:17.492 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:53:17.492 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:17.492 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:17.493 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:53:17.493 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:17.493 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:53:17.493 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:17.494 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:53:17.494 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:17.494 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:17.494 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:17.495 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:17.495 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:53:17.495 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:17.495 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:17.495 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T07:53:17.496 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:17.496 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:17.496 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:17.497 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T07:53:17.497 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:17.497 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:17.498 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:17.498 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:17.498 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:53:17.498 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:17.499 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:17.499 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:53:17.499 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:17.500 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:17.500 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:17.500 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:53:17.501 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:17.501 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:17.501 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:17.501 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:17.501 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:53:17.502 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:17.502 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:17.502 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:53:17.502 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:17.503 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:17.503 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:17.503 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:53:17.503 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:17.504 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:17.504 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:17.504 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:53:17.504 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:17.504 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:53:17.505 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:53:17.505 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:53:17.505 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:17.505 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:53:17.507 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.910082_164.1 2021-09-02T07:53:22.509 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:53:22.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:53:22.511 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:53:22.581 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:53:22.581 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:53:23.075 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:53:23.076 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:53:23.076 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:53:23.076 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:53:23.077 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:53:23.077 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:53:23.077 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:23.077 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:23.078 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:53:23.078 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:23.078 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:53:23.078 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:23.079 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:23.079 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:53:23.079 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:23.079 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:53:23.079 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:23.080 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:53:23.080 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:23.080 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:23.080 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:23.081 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:23.081 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:53:23.081 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:23.081 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:23.082 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.990121_170.1", 2021-09-02T07:53:23.082 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:23.082 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:23.082 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:23.083 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.990121Z", 2021-09-02T07:53:23.083 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:23.083 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:23.083 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:23.084 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:23.084 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:53:23.084 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:23.084 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:23.084 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:53:23.085 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:23.085 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:23.085 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:23.085 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:53:23.086 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:23.086 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:23.086 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:23.086 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:23.087 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:53:23.087 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:23.087 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:23.087 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:53:23.087 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:23.088 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:23.088 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:23.088 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:53:23.088 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:23.089 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:23.089 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:23.089 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:53:23.089 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:23.089 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:53:23.090 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:53:23.090 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:53:23.090 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:23.091 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:53:23.092 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.990121_170.1 2021-09-02T07:53:28.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:53:28.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:53:28.094 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:53:28.148 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:53:28.148 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:53:28.323 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:53:28.325 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:53:28.325 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:53:28.325 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:53:28.325 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:53:28.326 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:53:28.326 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:28.326 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:28.326 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:53:28.327 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:28.327 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:53:28.327 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:28.327 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:28.328 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:53:28.328 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:28.329 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:53:28.329 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:28.329 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:53:28.329 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:28.330 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:28.330 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:28.330 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:28.330 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:53:28.331 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:28.331 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:28.331 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.844214_158.1", 2021-09-02T07:53:28.331 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:28.331 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:28.332 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:28.332 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.844214Z", 2021-09-02T07:53:28.332 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:28.332 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:28.333 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:28.333 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:28.333 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:53:28.333 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:28.334 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:28.335 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:53:28.336 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:28.336 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:28.337 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:28.338 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:53:28.338 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:28.339 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:28.339 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:28.339 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:28.340 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:53:28.340 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:28.341 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:28.341 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:53:28.341 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:28.342 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:28.342 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:28.342 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:53:28.343 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:28.343 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:28.343 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:28.344 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:53:28.344 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:28.344 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:53:28.345 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:53:28.345 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:53:28.346 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:28.346 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:53:28.349 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.844214_158.1 2021-09-02T07:53:33.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:53:33.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:53:33.352 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:53:33.401 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:53:33.402 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:53:33.487 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:53:33.487 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:53:33.487 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:53:33.488 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:53:33.488 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:53:33.488 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:53:33.489 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:33.489 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:33.489 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:53:33.489 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:33.490 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:53:33.490 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:33.490 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:33.490 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:53:33.491 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:33.491 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:53:33.491 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:33.491 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:53:33.491 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:33.492 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:33.492 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:33.492 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:33.492 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:53:33.493 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:33.493 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:33.493 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.867814_160.1", 2021-09-02T07:53:33.493 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:33.493 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:33.494 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:33.494 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.867814Z", 2021-09-02T07:53:33.494 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:33.494 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:33.495 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:33.495 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:33.495 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:53:33.495 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:33.495 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:33.496 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:53:33.496 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:33.496 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:33.496 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:33.497 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:53:33.497 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:33.497 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:33.497 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:33.497 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:33.498 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:53:33.498 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:33.498 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:33.498 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:53:33.499 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:33.499 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:33.499 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:33.499 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:53:33.499 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:33.500 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:33.500 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:33.500 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:53:33.500 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:33.501 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:53:33.501 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:53:33.501 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:53:33.501 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:33.501 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:53:33.503 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.867814_160.1 2021-09-02T07:53:38.504 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:53:38.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:53:38.506 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:53:38.552 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:53:38.552 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:53:38.621 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:53:38.622 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:53:38.623 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:53:38.623 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:53:38.624 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:53:38.625 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:53:38.625 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:38.626 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:38.626 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:53:38.627 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:38.628 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:53:38.628 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:38.629 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:38.630 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:53:38.630 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:38.631 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:53:38.631 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:38.631 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:53:38.632 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:38.632 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:38.633 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:38.633 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:38.634 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:53:38.634 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:38.635 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:38.635 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.844214_158.1", 2021-09-02T07:53:38.636 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:38.637 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:38.637 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:38.638 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.844214Z", 2021-09-02T07:53:38.639 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:38.639 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:38.640 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:38.640 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:38.641 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:53:38.641 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:38.642 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:38.642 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:53:38.643 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:38.644 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:38.644 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:38.645 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:53:38.646 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:38.646 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:38.646 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:38.647 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:38.647 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:53:38.648 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:38.649 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:38.649 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:53:38.651 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:38.651 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:38.651 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:38.652 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:53:38.653 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:38.653 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:38.653 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:38.653 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:53:38.654 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:38.654 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:53:38.654 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:53:38.654 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:53:38.655 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:38.655 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:53:38.656 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.844214_158.1 2021-09-02T07:53:43.658 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:53:43.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:53:43.659 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:53:43.753 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:53:43.755 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:53:43.815 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:53:43.815 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:53:43.816 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:53:43.817 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:53:43.817 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:53:43.818 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:53:43.818 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:43.818 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:43.819 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:53:43.819 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:43.819 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:53:43.820 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:43.820 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:43.821 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:53:43.821 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:43.821 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:53:43.822 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:43.822 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:53:43.823 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:43.823 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:43.823 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:43.824 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:43.824 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:53:43.824 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:43.824 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:43.825 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.862512_159.1", 2021-09-02T07:53:43.825 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:43.825 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:43.826 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:43.827 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.862512Z", 2021-09-02T07:53:43.827 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:43.827 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:43.828 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:43.828 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:43.828 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:53:43.829 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:43.829 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:43.829 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:53:43.830 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:43.830 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:43.830 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:43.831 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:53:43.831 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:43.832 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:43.832 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:43.832 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:43.832 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:53:43.833 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:43.833 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:43.833 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:53:43.835 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:43.835 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:43.835 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:43.836 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:53:43.837 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:43.837 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:43.837 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:43.838 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:53:43.838 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:43.839 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:53:43.839 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:53:43.839 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:53:43.840 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:43.840 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:53:43.843 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.862512_159.1 2021-09-02T07:53:48.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:53:48.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:53:48.846 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:53:48.940 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:53:48.940 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:53:49.294 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:53:49.294 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:53:49.294 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:53:49.295 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:53:49.295 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:53:49.295 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:53:49.295 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:49.296 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:49.296 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:53:49.296 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:49.296 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:53:49.297 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:49.297 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:49.297 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:53:49.298 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:49.298 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:53:49.298 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:49.298 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:53:49.299 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:49.299 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:49.299 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:49.299 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:49.299 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:53:49.300 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:49.300 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:49.300 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.929348_167.1", 2021-09-02T07:53:49.300 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:49.301 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:49.301 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:49.301 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.929348Z", 2021-09-02T07:53:49.301 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:49.301 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:49.302 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:49.302 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:49.302 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:53:49.302 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:49.303 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:49.303 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:53:49.303 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:49.303 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:49.304 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:49.304 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:53:49.304 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:49.305 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:49.305 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:49.305 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:49.305 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:53:49.305 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:49.306 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:53:49.306 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:53:49.306 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:53:49.306 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:53:49.307 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:53:49.308 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:53:49.308 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:49.308 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:49.308 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:49.309 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:53:49.309 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:49.309 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:53:49.309 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:53:49.310 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:53:49.311 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:49.311 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:53:49.312 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630568906.235201_264.1 target=1_1630568126.929348_167.1 2021-09-02T07:53:54.314 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2021-09-02T07:53:54.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:53:54.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:53:54.317 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-02T07:53:54.360 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:53:54.360 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:53:54.386 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:53:54.386+0000 7fe6b835e380 0 failed reading obj info from .rgw.root:zone_info.5bdffc94-40ec-4bcf-a75a-819f7216b883: (2) No such file or directory 2021-09-02T07:53:54.387 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:53:54.386+0000 7fe6b835e380 0 WARNING: could not read zone params for zone id=5bdffc94-40ec-4bcf-a75a-819f7216b883 name=test-zone3 2021-09-02T07:53:54.387 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:53:54.386+0000 7fe6b835e380 0 failed reading obj info from .rgw.root:zone_info.8b6c886f-f9c4-4c5c-b256-edd3d1362f1c: (2) No such file or directory 2021-09-02T07:53:54.387 INFO:teuthology.orchestra.run.smithi197.stderr:2021-09-02T07:53:54.386+0000 7fe6b835e380 0 WARNING: could not read zone params for zone id=8b6c886f-f9c4-4c5c-b256-edd3d1362f1c name=test-zone1 2021-09-02T07:53:54.392 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:53:54.392 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "e43b32ce-d30a-498d-8211-34e9897b4509", 2021-09-02T07:53:54.392 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "test-zone2", 2021-09-02T07:53:54.393 INFO:teuthology.orchestra.run.smithi197.stdout: "domain_root": "test-zone2.rgw.meta:root", 2021-09-02T07:53:54.393 INFO:teuthology.orchestra.run.smithi197.stdout: "control_pool": "test-zone2.rgw.control", 2021-09-02T07:53:54.393 INFO:teuthology.orchestra.run.smithi197.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2021-09-02T07:53:54.393 INFO:teuthology.orchestra.run.smithi197.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2021-09-02T07:53:54.394 INFO:teuthology.orchestra.run.smithi197.stdout: "log_pool": "test-zone2.rgw.log", 2021-09-02T07:53:54.394 INFO:teuthology.orchestra.run.smithi197.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2021-09-02T07:53:54.394 INFO:teuthology.orchestra.run.smithi197.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2021-09-02T07:53:54.394 INFO:teuthology.orchestra.run.smithi197.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2021-09-02T07:53:54.395 INFO:teuthology.orchestra.run.smithi197.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2021-09-02T07:53:54.395 INFO:teuthology.orchestra.run.smithi197.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2021-09-02T07:53:54.395 INFO:teuthology.orchestra.run.smithi197.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2021-09-02T07:53:54.395 INFO:teuthology.orchestra.run.smithi197.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2021-09-02T07:53:54.396 INFO:teuthology.orchestra.run.smithi197.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2021-09-02T07:53:54.396 INFO:teuthology.orchestra.run.smithi197.stdout: "otp_pool": "test-zone2.rgw.otp", 2021-09-02T07:53:54.396 INFO:teuthology.orchestra.run.smithi197.stdout: "system_key": { 2021-09-02T07:53:54.397 INFO:teuthology.orchestra.run.smithi197.stdout: "access_key": "YDUXR4Y826KQKR5A", 2021-09-02T07:53:54.397 INFO:teuthology.orchestra.run.smithi197.stdout: "secret_key": "DglJ1ILvyGabxYVcMp6FQYY5m2LbRf2w" 2021-09-02T07:53:54.397 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:54.397 INFO:teuthology.orchestra.run.smithi197.stdout: "placement_pools": [ 2021-09-02T07:53:54.398 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:54.398 INFO:teuthology.orchestra.run.smithi197.stdout: "key": "default-placement", 2021-09-02T07:53:54.398 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:53:54.398 INFO:teuthology.orchestra.run.smithi197.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2021-09-02T07:53:54.398 INFO:teuthology.orchestra.run.smithi197.stdout: "storage_classes": { 2021-09-02T07:53:54.399 INFO:teuthology.orchestra.run.smithi197.stdout: "STANDARD": { 2021-09-02T07:53:54.399 INFO:teuthology.orchestra.run.smithi197.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2021-09-02T07:53:54.399 INFO:teuthology.orchestra.run.smithi197.stdout: "compression_type": "random" 2021-09-02T07:53:54.399 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:54.400 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:54.400 INFO:teuthology.orchestra.run.smithi197.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2021-09-02T07:53:54.400 INFO:teuthology.orchestra.run.smithi197.stdout: "index_type": 0 2021-09-02T07:53:54.400 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:54.400 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:54.401 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:53:54.401 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T07:53:54.402 INFO:teuthology.orchestra.run.smithi197.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2021-09-02T07:53:54.402 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:53:54.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:53:54.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T07:53:54.405 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-02T07:53:54.490 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:53:54.490 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:53:54.644 INFO:teuthology.orchestra.run.smithi197.stderr:Sending period to new master zone 8b6c886f-f9c4-4c5c-b256-edd3d1362f1c 2021-09-02T07:53:55.686 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:53:55.687 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:53:55.687 INFO:teuthology.orchestra.run.smithi197.stdout: "epoch": 5, 2021-09-02T07:53:55.687 INFO:teuthology.orchestra.run.smithi197.stdout: "predecessor_uuid": "dcf62b62-291a-46a6-867e-5263b7775f6a", 2021-09-02T07:53:55.687 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": [], 2021-09-02T07:53:55.688 INFO:teuthology.orchestra.run.smithi197.stdout: "period_map": { 2021-09-02T07:53:55.688 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:53:55.688 INFO:teuthology.orchestra.run.smithi197.stdout: "zonegroups": [ 2021-09-02T07:53:55.689 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:55.689 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "test-zonegroup", 2021-09-02T07:53:55.689 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "test-zonegroup", 2021-09-02T07:53:55.689 INFO:teuthology.orchestra.run.smithi197.stdout: "api_name": "", 2021-09-02T07:53:55.689 INFO:teuthology.orchestra.run.smithi197.stdout: "is_master": "true", 2021-09-02T07:53:55.690 INFO:teuthology.orchestra.run.smithi197.stdout: "endpoints": [ 2021-09-02T07:53:55.690 INFO:teuthology.orchestra.run.smithi197.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2021-09-02T07:53:55.690 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:53:55.690 INFO:teuthology.orchestra.run.smithi197.stdout: "hostnames": [], 2021-09-02T07:53:55.691 INFO:teuthology.orchestra.run.smithi197.stdout: "hostnames_s3website": [], 2021-09-02T07:53:55.691 INFO:teuthology.orchestra.run.smithi197.stdout: "master_zone": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:53:55.691 INFO:teuthology.orchestra.run.smithi197.stdout: "zones": [ 2021-09-02T07:53:55.691 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:55.691 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "5bdffc94-40ec-4bcf-a75a-819f7216b883", 2021-09-02T07:53:55.692 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "test-zone3", 2021-09-02T07:53:55.692 INFO:teuthology.orchestra.run.smithi197.stdout: "endpoints": [ 2021-09-02T07:53:55.692 INFO:teuthology.orchestra.run.smithi197.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2021-09-02T07:53:55.692 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:53:55.693 INFO:teuthology.orchestra.run.smithi197.stdout: "log_meta": "false", 2021-09-02T07:53:55.693 INFO:teuthology.orchestra.run.smithi197.stdout: "log_data": "true", 2021-09-02T07:53:55.693 INFO:teuthology.orchestra.run.smithi197.stdout: "bucket_index_max_shards": 11, 2021-09-02T07:53:55.693 INFO:teuthology.orchestra.run.smithi197.stdout: "read_only": "false", 2021-09-02T07:53:55.693 INFO:teuthology.orchestra.run.smithi197.stdout: "tier_type": "", 2021-09-02T07:53:55.694 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_from_all": "true", 2021-09-02T07:53:55.694 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_from": [], 2021-09-02T07:53:55.694 INFO:teuthology.orchestra.run.smithi197.stdout: "redirect_zone": "" 2021-09-02T07:53:55.694 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:55.695 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:55.695 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:53:55.695 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "test-zone1", 2021-09-02T07:53:55.695 INFO:teuthology.orchestra.run.smithi197.stdout: "endpoints": [ 2021-09-02T07:53:55.696 INFO:teuthology.orchestra.run.smithi197.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2021-09-02T07:53:55.696 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:53:55.696 INFO:teuthology.orchestra.run.smithi197.stdout: "log_meta": "false", 2021-09-02T07:53:55.696 INFO:teuthology.orchestra.run.smithi197.stdout: "log_data": "true", 2021-09-02T07:53:55.697 INFO:teuthology.orchestra.run.smithi197.stdout: "bucket_index_max_shards": 11, 2021-09-02T07:53:55.697 INFO:teuthology.orchestra.run.smithi197.stdout: "read_only": "false", 2021-09-02T07:53:55.697 INFO:teuthology.orchestra.run.smithi197.stdout: "tier_type": "", 2021-09-02T07:53:55.698 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_from_all": "true", 2021-09-02T07:53:55.698 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_from": [], 2021-09-02T07:53:55.698 INFO:teuthology.orchestra.run.smithi197.stdout: "redirect_zone": "" 2021-09-02T07:53:55.699 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:55.699 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:55.699 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "a0dd8689-bd06-4f6e-8710-337db82a68b6", 2021-09-02T07:53:55.699 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "test-zone4", 2021-09-02T07:53:55.700 INFO:teuthology.orchestra.run.smithi197.stdout: "endpoints": [ 2021-09-02T07:53:55.700 INFO:teuthology.orchestra.run.smithi197.stdout: "http://smithi197.front.sepia.ceph.com:8001" 2021-09-02T07:53:55.700 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:53:55.700 INFO:teuthology.orchestra.run.smithi197.stdout: "log_meta": "false", 2021-09-02T07:53:55.701 INFO:teuthology.orchestra.run.smithi197.stdout: "log_data": "true", 2021-09-02T07:53:55.701 INFO:teuthology.orchestra.run.smithi197.stdout: "bucket_index_max_shards": 11, 2021-09-02T07:53:55.701 INFO:teuthology.orchestra.run.smithi197.stdout: "read_only": "false", 2021-09-02T07:53:55.701 INFO:teuthology.orchestra.run.smithi197.stdout: "tier_type": "pubsub", 2021-09-02T07:53:55.701 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_from_all": "false", 2021-09-02T07:53:55.702 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_from": [ 2021-09-02T07:53:55.702 INFO:teuthology.orchestra.run.smithi197.stdout: "test-zone1" 2021-09-02T07:53:55.702 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:53:55.702 INFO:teuthology.orchestra.run.smithi197.stdout: "redirect_zone": "" 2021-09-02T07:53:55.703 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:55.703 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:55.703 INFO:teuthology.orchestra.run.smithi197.stdout: "id": "e43b32ce-d30a-498d-8211-34e9897b4509", 2021-09-02T07:53:55.703 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "test-zone2", 2021-09-02T07:53:55.704 INFO:teuthology.orchestra.run.smithi197.stdout: "endpoints": [ 2021-09-02T07:53:55.704 INFO:teuthology.orchestra.run.smithi197.stdout: "http://smithi197.front.sepia.ceph.com:8000" 2021-09-02T07:53:55.704 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:53:55.704 INFO:teuthology.orchestra.run.smithi197.stdout: "log_meta": "false", 2021-09-02T07:53:55.705 INFO:teuthology.orchestra.run.smithi197.stdout: "log_data": "true", 2021-09-02T07:53:55.705 INFO:teuthology.orchestra.run.smithi197.stdout: "bucket_index_max_shards": 11, 2021-09-02T07:53:55.705 INFO:teuthology.orchestra.run.smithi197.stdout: "read_only": "false", 2021-09-02T07:53:55.705 INFO:teuthology.orchestra.run.smithi197.stdout: "tier_type": "", 2021-09-02T07:53:55.706 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_from_all": "false", 2021-09-02T07:53:55.706 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_from": [], 2021-09-02T07:53:55.706 INFO:teuthology.orchestra.run.smithi197.stdout: "redirect_zone": "" 2021-09-02T07:53:55.706 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:55.707 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:53:55.707 INFO:teuthology.orchestra.run.smithi197.stdout: "placement_targets": [ 2021-09-02T07:53:55.707 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:55.708 INFO:teuthology.orchestra.run.smithi197.stdout: "name": "default-placement", 2021-09-02T07:53:55.708 INFO:teuthology.orchestra.run.smithi197.stdout: "tags": [], 2021-09-02T07:53:55.708 INFO:teuthology.orchestra.run.smithi197.stdout: "storage_classes": [ 2021-09-02T07:53:55.708 INFO:teuthology.orchestra.run.smithi197.stdout: "STANDARD" 2021-09-02T07:53:55.709 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:53:55.709 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:55.709 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:53:55.709 INFO:teuthology.orchestra.run.smithi197.stdout: "default_placement": "default-placement", 2021-09-02T07:53:55.710 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T07:53:55.710 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_policy": { 2021-09-02T07:53:55.710 INFO:teuthology.orchestra.run.smithi197.stdout: "groups": [] 2021-09-02T07:53:55.710 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:55.711 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:55.711 INFO:teuthology.orchestra.run.smithi197.stdout: ], 2021-09-02T07:53:55.711 INFO:teuthology.orchestra.run.smithi197.stdout: "short_zone_ids": [ 2021-09-02T07:53:55.711 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:55.712 INFO:teuthology.orchestra.run.smithi197.stdout: "key": "5bdffc94-40ec-4bcf-a75a-819f7216b883", 2021-09-02T07:53:55.713 INFO:teuthology.orchestra.run.smithi197.stdout: "val": 3291489215 2021-09-02T07:53:55.713 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:55.713 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:55.713 INFO:teuthology.orchestra.run.smithi197.stdout: "key": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:53:55.714 INFO:teuthology.orchestra.run.smithi197.stdout: "val": 2303553998 2021-09-02T07:53:55.714 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:55.714 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:55.714 INFO:teuthology.orchestra.run.smithi197.stdout: "key": "a0dd8689-bd06-4f6e-8710-337db82a68b6", 2021-09-02T07:53:55.715 INFO:teuthology.orchestra.run.smithi197.stdout: "val": 3813003288 2021-09-02T07:53:55.715 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:55.715 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:53:55.716 INFO:teuthology.orchestra.run.smithi197.stdout: "key": "e43b32ce-d30a-498d-8211-34e9897b4509", 2021-09-02T07:53:55.716 INFO:teuthology.orchestra.run.smithi197.stdout: "val": 1397540283 2021-09-02T07:53:55.716 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:55.716 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:53:55.717 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:55.717 INFO:teuthology.orchestra.run.smithi197.stdout: "master_zonegroup": "test-zonegroup", 2021-09-02T07:53:55.718 INFO:teuthology.orchestra.run.smithi197.stdout: "master_zone": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T07:53:55.718 INFO:teuthology.orchestra.run.smithi197.stdout: "period_config": { 2021-09-02T07:53:55.718 INFO:teuthology.orchestra.run.smithi197.stdout: "bucket_quota": { 2021-09-02T07:53:55.718 INFO:teuthology.orchestra.run.smithi197.stdout: "enabled": false, 2021-09-02T07:53:55.718 INFO:teuthology.orchestra.run.smithi197.stdout: "check_on_raw": false, 2021-09-02T07:53:55.719 INFO:teuthology.orchestra.run.smithi197.stdout: "max_size": -1, 2021-09-02T07:53:55.719 INFO:teuthology.orchestra.run.smithi197.stdout: "max_size_kb": 0, 2021-09-02T07:53:55.720 INFO:teuthology.orchestra.run.smithi197.stdout: "max_objects": -1 2021-09-02T07:53:55.720 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:55.720 INFO:teuthology.orchestra.run.smithi197.stdout: "user_quota": { 2021-09-02T07:53:55.721 INFO:teuthology.orchestra.run.smithi197.stdout: "enabled": false, 2021-09-02T07:53:55.722 INFO:teuthology.orchestra.run.smithi197.stdout: "check_on_raw": false, 2021-09-02T07:53:55.722 INFO:teuthology.orchestra.run.smithi197.stdout: "max_size": -1, 2021-09-02T07:53:55.722 INFO:teuthology.orchestra.run.smithi197.stdout: "max_size_kb": 0, 2021-09-02T07:53:55.723 INFO:teuthology.orchestra.run.smithi197.stdout: "max_objects": -1 2021-09-02T07:53:55.723 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:53:55.723 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:53:55.723 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T07:53:55.724 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_name": "test-realm", 2021-09-02T07:53:55.724 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:53:55.724 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:53:55.724 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2021-09-02T07:54:55.726 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-45 2021-09-02T07:54:55.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:54:55.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:54:55.813 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:54:55.867 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:54:55.867 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:54:55.918 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T07:54:55.934 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T07:54:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:54:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:54:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:32:10.734248Z" 2021-09-02T07:54:55.935 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:54:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:54:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630569295.762188_292.1", 2021-09-02T07:54:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:54:55.762188Z" 2021-09-02T07:54:55.936 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:54:55.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:54:55.937 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:54:55.937 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:54:55.937 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:54:55.937 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:54:55.938 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:54:55.938 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T07:54:55.938 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:54:55.939 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T07:54:55.940 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T07:54:55.941 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:54:55.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:54:55.942 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:54:55.999 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:54:56.000 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:54:56.238 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:54:56.238 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:54:56.239 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:54:56.239 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:54:56.240 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:54:56.240 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:54:56.241 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:54:56.241 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:54:56.241 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:54:56.242 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:54:56.242 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:54:56.243 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:54:56.244 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:54:56.244 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:54:56.245 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:54:56.245 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:54:56.246 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:54:56.246 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:54:56.247 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:54:56.247 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:54:56.247 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:54:56.248 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:54:56.248 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:54:56.249 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:54:56.249 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:54:56.250 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568906.235201_264.1", 2021-09-02T07:54:56.250 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:54:56.251 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:54:56.251 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:54:56.251 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:48:26.235201Z", 2021-09-02T07:54:56.252 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:54:56.252 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:54:56.253 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:54:56.253 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:54:56.254 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:54:56.254 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:54:56.254 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:54:56.255 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:54:56.255 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:54:56.255 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:54:56.256 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:54:56.256 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:54:56.256 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:54:56.257 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:54:56.257 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:54:56.257 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:54:56.258 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:54:56.258 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:54:56.259 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:54:56.259 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:54:56.259 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:54:56.259 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:54:56.259 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:54:56.260 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:54:56.260 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:54:56.260 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:54:56.260 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:54:56.261 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:54:56.261 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:54:56.261 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:54:56.262 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:54:56.262 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:54:56.262 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:54:56.262 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:54:56.263 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568906.235201_264.1 2021-09-02T07:55:01.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:01.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:01.267 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:55:01.360 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:55:01.361 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:55:01.586 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:55:01.587 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:55:01.587 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:55:01.587 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:55:01.588 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:55:01.588 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:55:01.588 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:01.588 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:01.589 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:55:01.589 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:01.589 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:55:01.589 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:01.590 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:01.590 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:55:01.590 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:01.590 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:55:01.590 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:01.591 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:55:01.591 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:01.591 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:01.591 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:01.592 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:01.592 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:55:01.592 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:01.592 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:01.592 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568906.235201_264.1", 2021-09-02T07:55:01.593 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:01.593 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:01.593 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:01.593 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:48:26.235201Z", 2021-09-02T07:55:01.594 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:01.594 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:01.594 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:01.594 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:01.595 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:55:01.595 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:01.595 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:01.595 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:01.596 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:01.596 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:01.596 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:01.597 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:01.597 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:01.597 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:01.597 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:01.598 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:01.598 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:55:01.598 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:01.598 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:01.598 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:01.599 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:01.599 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:01.599 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:01.599 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:01.600 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:01.600 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:01.600 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:01.600 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:55:01.600 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:01.601 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:55:01.601 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:55:01.601 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:55:01.601 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:01.602 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:55:01.602 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568906.235201_264.1 2021-09-02T07:55:06.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:06.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:06.605 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:55:06.650 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:55:06.650 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:55:06.715 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:55:06.715 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:55:06.715 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:55:06.716 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:55:06.716 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:55:06.716 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:55:06.716 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:06.717 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:06.717 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:55:06.717 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:06.717 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:55:06.717 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:06.718 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:06.718 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:55:06.718 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:06.719 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:55:06.719 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:06.719 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:55:06.719 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:06.719 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:06.720 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:06.720 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:06.720 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:55:06.720 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:06.721 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:06.721 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568906.235201_264.1", 2021-09-02T07:55:06.721 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:06.721 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:06.722 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:06.722 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:48:26.235201Z", 2021-09-02T07:55:06.722 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:06.722 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:06.722 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:06.723 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:06.723 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:55:06.723 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:06.723 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:06.724 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:06.724 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:06.724 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:06.724 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:06.724 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:06.725 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:06.725 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:06.725 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:06.725 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:06.726 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:55:06.726 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:06.726 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:06.726 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:06.727 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:06.727 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:06.727 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:06.727 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:06.727 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:06.728 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:06.728 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:06.728 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:55:06.728 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:06.729 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:55:06.729 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:55:06.729 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:55:06.729 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:06.729 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:55:06.731 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568906.235201_264.1 2021-09-02T07:55:11.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:11.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:11.734 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:55:11.777 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:55:11.778 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:55:11.852 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:55:11.852 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:55:11.852 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:55:11.852 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:55:11.853 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:55:11.853 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:55:11.853 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:11.853 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:11.854 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:55:11.854 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:11.854 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:55:11.854 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:11.854 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:11.855 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:55:11.855 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:11.855 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:55:11.855 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:11.856 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:55:11.856 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:11.856 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:11.856 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:11.856 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:11.857 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:55:11.857 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:11.857 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:11.857 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568906.235201_264.1", 2021-09-02T07:55:11.857 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:11.858 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:11.858 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:11.858 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:48:26.235201Z", 2021-09-02T07:55:11.858 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:11.859 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:11.859 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:11.859 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:11.860 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:55:11.860 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:11.860 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:11.861 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:11.861 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:11.861 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:11.862 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:11.862 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:11.862 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:11.863 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:11.863 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:11.863 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:11.863 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:55:11.864 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:11.864 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:11.864 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:11.865 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:11.865 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:11.865 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:11.865 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:11.865 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:11.866 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:11.866 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:11.866 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:55:11.866 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:11.867 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:55:11.867 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:55:11.867 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:55:11.867 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:11.868 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:55:11.869 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568906.235201_264.1 2021-09-02T07:55:16.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:16.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:16.872 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:55:16.914 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:55:16.914 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:55:16.990 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:55:16.990 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:55:16.991 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:55:16.991 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:55:16.991 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:55:16.991 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:55:16.992 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:16.992 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:16.992 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:55:16.992 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:16.993 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:55:16.993 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:16.993 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:16.993 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:55:16.993 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:16.994 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:55:16.994 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:16.994 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:55:16.994 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:16.995 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:16.995 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:16.995 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:16.995 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:55:16.996 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:16.996 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:16.996 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568906.235201_264.1", 2021-09-02T07:55:16.997 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:16.997 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:16.997 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:16.997 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:48:26.235201Z", 2021-09-02T07:55:16.998 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:16.998 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:16.998 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:16.998 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:16.999 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:55:16.999 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:16.999 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:16.999 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:16.999 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:17.000 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:17.000 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:17.000 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:17.000 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:17.001 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:17.001 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:17.001 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:17.001 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:55:17.001 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:17.002 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:17.002 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:17.002 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:17.002 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:17.003 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:17.003 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:17.003 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:17.003 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:17.004 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:17.004 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:55:17.004 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:17.004 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:55:17.004 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:55:17.005 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:55:17.005 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:17.005 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:55:17.007 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568906.235201_264.1 2021-09-02T07:55:22.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:22.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:22.009 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:55:22.085 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:55:22.085 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:55:22.262 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:55:22.262 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:55:22.262 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:55:22.263 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:55:22.263 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:55:22.263 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:55:22.264 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:22.265 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:22.265 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:55:22.265 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:22.265 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:55:22.266 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:22.266 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:22.266 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:55:22.266 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:22.268 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:55:22.269 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:22.269 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:55:22.270 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:22.270 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:22.271 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:22.271 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:22.272 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:55:22.272 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:22.272 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:22.272 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568906.235201_264.1", 2021-09-02T07:55:22.273 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:22.274 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:22.274 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:22.276 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:48:26.235201Z", 2021-09-02T07:55:22.276 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:22.276 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:22.276 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:22.277 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:22.277 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:55:22.277 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:22.277 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:22.278 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:22.278 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:22.278 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:22.278 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:22.279 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:22.279 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:22.279 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:22.279 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:22.280 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:22.280 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:55:22.280 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:22.280 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:22.281 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:22.281 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:22.281 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:22.281 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:22.281 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:22.282 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:22.282 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:22.282 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:22.282 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:55:22.283 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:22.283 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:55:22.283 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:55:22.283 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:55:22.284 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:22.284 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:55:22.286 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568906.235201_264.1 2021-09-02T07:55:27.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:27.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:27.289 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:55:27.346 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:55:27.346 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:55:27.566 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:55:27.566 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:55:27.567 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:55:27.567 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:55:27.567 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:55:27.567 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:55:27.567 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:27.568 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:27.568 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:55:27.568 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:27.568 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:55:27.569 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:27.569 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:27.569 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:55:27.569 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:27.570 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:55:27.570 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:27.570 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:55:27.571 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:27.572 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:27.572 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:27.573 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:27.573 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:55:27.573 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:27.573 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:27.574 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568906.235201_264.1", 2021-09-02T07:55:27.574 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:27.574 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:27.574 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:27.575 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:48:26.235201Z", 2021-09-02T07:55:27.575 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:27.575 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:27.575 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:27.575 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:27.576 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:55:27.576 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:27.576 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:27.577 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:27.577 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:27.577 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:27.577 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:27.577 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:27.578 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:27.578 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:27.578 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:27.579 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:27.579 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:55:27.579 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:27.579 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:27.580 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:27.580 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:27.580 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:27.580 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:27.580 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:27.581 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:27.581 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:27.581 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:27.582 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:55:27.582 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:27.582 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:55:27.582 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:55:27.583 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:55:27.583 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:27.583 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:55:27.584 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568906.235201_264.1 2021-09-02T07:55:32.586 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:32.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:32.588 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:55:32.680 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:55:32.680 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:55:32.935 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:55:32.935 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:55:32.936 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:55:32.936 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:55:32.937 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:55:32.937 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:55:32.937 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:32.937 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:32.938 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:55:32.938 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:32.938 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:55:32.938 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:32.939 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:32.939 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:55:32.939 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:32.940 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:55:32.940 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:32.940 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:55:32.940 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:32.941 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:32.941 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:32.941 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:32.941 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:55:32.941 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:32.942 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:32.942 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568906.235201_264.1", 2021-09-02T07:55:32.942 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:32.942 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:32.943 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:32.943 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:48:26.235201Z", 2021-09-02T07:55:32.943 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:32.943 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:32.944 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:32.944 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:32.944 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:55:32.944 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:32.945 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:32.945 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:32.945 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:32.945 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:32.946 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:32.946 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:32.947 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:32.947 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:32.947 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:32.947 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:32.948 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:55:32.948 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:32.948 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:32.948 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:32.949 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:32.949 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:32.949 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:32.949 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:32.949 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:32.950 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:32.950 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:32.951 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:55:32.951 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:32.951 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:55:32.951 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:55:32.951 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:55:32.952 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:32.952 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:55:32.952 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568906.235201_264.1 2021-09-02T07:55:37.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:37.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:37.955 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:55:38.005 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:55:38.005 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:55:38.425 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:55:38.425 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:55:38.425 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:55:38.426 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:55:38.426 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:55:38.426 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:55:38.426 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:38.427 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:38.427 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:55:38.427 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:38.427 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:55:38.428 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:38.428 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:38.428 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:55:38.428 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:38.428 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:55:38.429 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:38.429 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:55:38.430 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:38.430 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:38.430 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:38.430 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:38.431 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:55:38.431 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:38.431 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:38.431 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568906.235201_264.1", 2021-09-02T07:55:38.432 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:38.432 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:38.432 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:38.433 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:48:26.235201Z", 2021-09-02T07:55:38.433 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:38.434 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:38.434 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:38.434 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:38.434 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:55:38.435 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:38.435 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:38.435 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:38.435 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:38.435 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:38.436 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:38.436 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:38.436 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:38.437 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:38.437 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:38.437 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:38.438 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:55:38.438 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:38.438 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:38.438 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:38.439 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:38.439 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:38.439 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:38.439 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:38.439 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:38.440 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:38.440 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:38.441 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:55:38.441 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:38.441 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:55:38.441 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:55:38.442 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:55:38.442 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:38.442 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:55:38.443 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568906.235201_264.1 2021-09-02T07:55:43.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:43.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:43.446 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:55:43.493 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:55:43.493 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:55:43.563 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:55:43.565 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:55:43.565 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:55:43.566 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:55:43.566 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:55:43.567 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:55:43.567 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:43.568 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:43.568 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:55:43.569 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:43.569 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:55:43.569 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:43.570 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:43.570 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:55:43.571 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:43.571 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:55:43.572 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:43.572 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:55:43.573 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:43.574 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:43.574 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:43.575 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:43.575 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:55:43.576 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:43.576 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:43.577 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568906.235201_264.1", 2021-09-02T07:55:43.577 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:43.578 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:43.578 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:43.578 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:48:26.235201Z", 2021-09-02T07:55:43.579 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:43.579 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:43.579 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:43.580 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:43.580 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:55:43.580 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:43.580 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:43.581 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:43.581 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:43.581 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:43.582 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:43.582 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:43.582 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:43.582 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:43.583 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:43.583 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:43.583 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:55:43.583 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:43.584 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:43.584 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:43.584 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:43.585 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:43.585 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:43.585 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:43.585 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:43.586 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:43.586 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:43.586 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:55:43.587 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:43.587 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:55:43.588 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:55:43.588 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:55:43.588 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:43.588 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:55:43.590 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568906.235201_264.1 2021-09-02T07:55:48.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:48.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:48.593 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:55:48.638 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:55:48.639 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:55:48.708 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:55:48.709 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:55:48.709 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:55:48.709 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:55:48.709 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:55:48.710 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:55:48.710 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:48.710 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:48.710 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:55:48.711 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:48.711 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:55:48.711 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:48.711 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:48.711 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:55:48.712 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:48.712 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:55:48.712 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:48.713 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:55:48.713 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:48.713 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:48.713 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:48.713 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:48.714 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:55:48.714 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:48.714 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:48.714 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568906.235201_264.1", 2021-09-02T07:55:48.715 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:48.715 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:48.715 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:48.715 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:48:26.235201Z", 2021-09-02T07:55:48.716 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:48.716 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:48.716 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:48.716 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:48.717 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:55:48.717 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:48.718 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:48.720 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:48.720 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:48.720 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:48.721 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:48.721 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:48.721 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:48.721 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:48.722 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:48.722 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:48.722 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:55:48.723 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:48.723 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:48.723 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:48.723 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:48.723 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:48.724 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:48.724 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:48.724 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:48.724 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:48.725 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:48.725 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:55:48.725 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:48.725 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:55:48.725 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:55:48.726 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:55:48.726 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:48.726 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:55:48.728 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568906.235201_264.1 2021-09-02T07:55:53.728 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:53.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:53.730 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:55:53.809 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:55:53.810 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:55:53.909 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:55:53.909 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:55:53.910 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:55:53.910 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:55:53.911 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:55:53.912 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:55:53.912 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:53.912 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:53.912 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:55:53.912 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:53.913 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:55:53.913 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:53.913 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:53.913 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:55:53.914 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:53.914 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:55:53.914 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:53.914 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:55:53.915 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:53.915 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:53.915 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:53.915 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:53.916 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:55:53.916 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:53.916 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:53.916 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568906.235201_264.1", 2021-09-02T07:55:53.916 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:53.917 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:53.917 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:53.917 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:48:26.235201Z", 2021-09-02T07:55:53.917 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:53.918 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:53.918 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:53.918 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:53.918 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:55:53.919 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:53.919 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:53.919 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:53.919 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:53.919 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:53.920 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:53.920 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:53.920 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:53.921 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:53.921 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:53.921 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:53.921 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:55:53.922 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:53.922 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:53.922 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:53.922 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:53.923 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:53.923 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:53.923 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:53.923 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:53.923 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:53.924 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:53.924 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:55:53.924 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:53.924 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:55:53.925 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:55:53.925 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:55:53.925 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:53.925 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:55:53.927 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568906.235201_264.1 2021-09-02T07:55:55.924 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2021-09-02T07:55:55.925+0000 7f042df10700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-02T07:55:58.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:58.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:55:58.929 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:55:58.997 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:55:58.997 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:55:59.323 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:55:59.323 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:55:59.323 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:55:59.323 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:55:59.326 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:55:59.327 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:55:59.327 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:59.327 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:59.328 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:55:59.328 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:59.328 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:55:59.328 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:59.329 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:59.329 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:55:59.329 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:59.329 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:55:59.329 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:59.330 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:55:59.330 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:59.330 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:59.330 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:59.331 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:59.331 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:55:59.331 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:59.331 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:59.332 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630569295.749456_284.1", 2021-09-02T07:55:59.332 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:59.332 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:59.332 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:59.333 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:54:55.749456Z", 2021-09-02T07:55:59.333 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:59.333 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:59.333 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:59.333 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:59.334 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:55:59.334 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:59.334 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:59.334 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:59.335 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:59.335 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:59.335 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:59.335 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:59.335 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:59.336 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:59.336 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:59.336 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:55:59.336 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:55:59.337 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:55:59.337 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:55:59.337 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:55:59.337 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:55:59.338 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:55:59.338 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:55:59.338 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:55:59.338 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:55:59.338 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:59.339 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:59.339 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:55:59.339 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:55:59.339 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:55:59.340 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:55:59.340 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:55:59.340 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:55:59.340 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:55:59.341 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630569295.749456_284.1 2021-09-02T07:56:04.342 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:04.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:04.346 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:56:04.450 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:56:04.450 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:56:04.633 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:56:04.633 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:56:04.634 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:56:04.634 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:56:04.634 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:56:04.634 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:56:04.635 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:04.635 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:04.635 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:56:04.635 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:04.635 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:56:04.636 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:04.636 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:04.636 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:56:04.636 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:04.637 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:56:04.637 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:04.637 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:56:04.637 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:04.638 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:04.638 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:04.638 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:04.638 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:56:04.638 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:04.639 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:04.639 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630569295.762188_292.1", 2021-09-02T07:56:04.639 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:04.639 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:04.641 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:04.641 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:54:55.762188Z", 2021-09-02T07:56:04.642 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:04.642 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:04.642 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:04.642 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:04.643 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:56:04.643 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:04.643 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:04.643 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:04.644 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:04.644 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:04.644 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:04.644 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:04.645 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:04.645 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:04.645 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:04.645 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:04.645 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:56:04.646 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:04.646 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:04.646 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:04.646 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:04.647 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:04.647 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:04.647 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:04.647 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:04.648 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:04.648 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:04.648 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:56:04.648 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:04.648 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:56:04.649 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:56:04.649 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:56:04.649 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:04.649 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:56:04.650 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T07:56:04.651 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T07:56:04.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:04.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:04.653 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:56:04.698 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:56:04.699 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:56:04.769 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T07:56:04.770 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T07:56:04.770 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T07:56:04.770 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T07:56:04.771 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T07:56:04.771 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:56:04.771 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:56:04.771 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:56:04.772 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T07:56:04.772 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:56:04.772 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T07:56:04.772 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:56:04.773 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:56:04.773 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:56:04.773 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:56:04.773 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T07:56:04.773 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:56:04.774 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:56:04.774 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:56:04.774 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:56:04.774 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:56:04.775 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:56:04.775 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T07:56:04.775 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:56:04.775 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:56:04.776 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630569295.762188_292.1", 2021-09-02T07:56:04.776 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:56:04.776 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:56:04.776 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:56:04.776 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:54:55.762188Z", 2021-09-02T07:56:04.777 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:56:04.777 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:56:04.777 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:56:04.777 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:56:04.778 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T07:56:04.778 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:56:04.778 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:56:04.778 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:56:04.779 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:56:04.779 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:56:04.779 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:56:04.779 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:56:04.779 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:56:04.780 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:56:04.780 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:56:04.780 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T07:56:04.780 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T07:56:04.781 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T07:56:04.781 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T07:56:04.781 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T07:56:04.781 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T07:56:04.782 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T07:56:04.782 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T07:56:04.782 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T07:56:04.782 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T07:56:04.783 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:56:04.783 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:56:04.783 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T07:56:04.783 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T07:56:04.783 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T07:56:04.784 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T07:56:04.784 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T07:56:04.784 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T07:56:04.784 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T07:56:04.786 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T07:56:04.786 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T07:56:04.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:04.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:04.788 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:56:04.847 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:56:04.848 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:56:05.275 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:56:05.275 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:56:05.276 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:56:05.276 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:56:05.276 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:56:05.276 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:56:05.277 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:05.277 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:05.277 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:56:05.277 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:05.277 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:56:05.278 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:05.278 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:05.278 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:56:05.278 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:05.279 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:56:05.279 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:05.279 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:56:05.279 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:05.279 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:05.280 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:05.280 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:05.280 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:56:05.280 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:05.281 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:05.281 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.985199_169.1", 2021-09-02T07:56:05.281 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:05.281 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:05.282 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:05.282 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.985199Z", 2021-09-02T07:56:05.282 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:05.282 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:05.283 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:05.283 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:05.283 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:56:05.283 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:05.284 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:05.284 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:05.284 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:05.284 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:05.284 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:05.285 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:05.285 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:05.285 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:05.285 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:05.286 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:05.286 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:56:05.286 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:05.286 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:05.287 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:05.287 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:05.287 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:05.288 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:05.288 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:05.288 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:05.289 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:05.289 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:05.289 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:56:05.289 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:05.289 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:56:05.290 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:56:05.290 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:56:05.290 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:05.291 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:56:05.292 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.985199_169.1 2021-09-02T07:56:10.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:10.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:10.295 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:56:10.348 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:56:10.349 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:56:10.425 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:56:10.426 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:56:10.426 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:56:10.426 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:56:10.427 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:56:10.427 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:56:10.427 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:10.427 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:10.428 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:56:10.428 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:10.428 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:56:10.428 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:10.429 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:10.429 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:56:10.429 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:10.429 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:56:10.430 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:10.431 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:56:10.431 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:10.431 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:10.431 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:10.432 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:10.432 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:56:10.432 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:10.432 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:10.433 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.039849_171.1", 2021-09-02T07:56:10.433 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:10.433 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:10.433 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:10.434 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.039849Z", 2021-09-02T07:56:10.434 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:10.434 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:10.434 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:10.434 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:10.435 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:56:10.435 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:10.435 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:10.435 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:10.436 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:10.436 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:10.436 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:10.436 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:10.437 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:10.437 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:10.437 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:10.437 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:10.438 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:56:10.438 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:10.438 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:10.438 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:10.438 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:10.439 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:10.439 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:10.439 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:10.439 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:10.440 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:10.440 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:10.440 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:56:10.440 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:10.441 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:56:10.441 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:56:10.441 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:56:10.441 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:10.441 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:56:10.443 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.039849_171.1 2021-09-02T07:56:15.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:15.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:15.445 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:56:15.529 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:56:15.530 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:56:15.943 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:56:15.943 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:56:15.943 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:56:15.944 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:56:15.944 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:56:15.944 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:56:15.944 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:15.945 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:15.945 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:56:15.945 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:15.945 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:56:15.946 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:15.946 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:15.946 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:56:15.946 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:15.947 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:56:15.947 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:15.947 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:56:15.947 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:15.948 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:15.948 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:15.948 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:15.948 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:56:15.949 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:15.949 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:15.949 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.929348_167.1", 2021-09-02T07:56:15.949 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:15.950 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:15.950 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:15.950 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.929348Z", 2021-09-02T07:56:15.950 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:15.950 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:15.951 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:15.951 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:15.951 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:56:15.951 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:15.952 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:15.952 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:15.952 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:15.952 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:15.953 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:15.953 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:15.953 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:15.953 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:15.953 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:15.954 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:15.954 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:56:15.954 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:15.954 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:15.955 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:15.955 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:15.955 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:15.955 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:15.956 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:15.956 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:15.956 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:15.956 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:15.956 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:56:15.957 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:15.957 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:56:15.957 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:56:15.958 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:56:15.958 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:15.958 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:56:15.960 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.929348_167.1 2021-09-02T07:56:20.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:20.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:20.962 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:56:21.009 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:56:21.010 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:56:21.086 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:56:21.086 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:56:21.086 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:56:21.087 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:56:21.087 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:56:21.087 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:56:21.087 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:21.088 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:21.088 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:56:21.088 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:21.088 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:56:21.089 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:21.089 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:21.089 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:56:21.090 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:21.090 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:56:21.090 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:21.090 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:56:21.091 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:21.091 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:21.091 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:21.091 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:21.091 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:56:21.092 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:21.092 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:21.092 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T07:56:21.092 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:21.093 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:21.093 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:21.093 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T07:56:21.093 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:21.094 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:21.094 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:21.094 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:21.094 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:56:21.094 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:21.095 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:21.095 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:21.095 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:21.096 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:21.096 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:21.096 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:21.096 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:21.097 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:21.097 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:21.097 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:21.098 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:56:21.098 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:21.098 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:21.099 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:21.099 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:21.099 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:21.099 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:21.099 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:21.100 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:21.100 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:21.100 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:21.101 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:56:21.101 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:21.101 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:56:21.101 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:56:21.102 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:56:21.102 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:21.102 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:56:21.103 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.103991_175.1 2021-09-02T07:56:26.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:26.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:26.106 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:56:26.155 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:56:26.155 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:56:26.232 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:56:26.232 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:56:26.233 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:56:26.233 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:56:26.233 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:56:26.233 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:56:26.234 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:26.234 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:26.234 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:56:26.234 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:26.235 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:56:26.235 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:26.235 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:26.235 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:56:26.236 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:26.236 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:56:26.236 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:26.237 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:56:26.237 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:26.237 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:26.238 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:26.238 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:26.238 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:56:26.239 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:26.239 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:26.239 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T07:56:26.239 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:26.240 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:26.240 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:26.240 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T07:56:26.240 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:26.241 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:26.241 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:26.241 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:26.242 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:56:26.242 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:26.242 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:26.243 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:26.243 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:26.243 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:26.244 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:26.244 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:26.245 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:26.245 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:26.246 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:26.247 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:26.247 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:56:26.247 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:26.248 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:26.248 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:26.249 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:26.249 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:26.250 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:26.250 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:26.250 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:26.251 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:26.251 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:26.252 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:56:26.252 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:26.252 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:56:26.253 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:56:26.253 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:56:26.254 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:26.255 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:56:26.257 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.875533_162.1 2021-09-02T07:56:31.259 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:31.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:31.260 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:56:31.313 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:56:31.314 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:56:31.400 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:56:31.401 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:56:31.401 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:56:31.401 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:56:31.402 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:56:31.402 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:56:31.402 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:31.402 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:31.403 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:56:31.403 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:31.403 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:56:31.403 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:31.403 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:31.404 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:56:31.404 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:31.404 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:56:31.404 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:31.405 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:56:31.405 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:31.405 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:31.405 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:31.406 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:31.406 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:56:31.406 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:31.406 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:31.406 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T07:56:31.407 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:31.407 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:31.407 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:31.407 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T07:56:31.408 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:31.408 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:31.408 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:31.408 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:31.409 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:56:31.409 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:31.409 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:31.409 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:31.410 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:31.410 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:31.410 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:31.411 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:31.411 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:31.411 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:31.411 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:31.412 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:31.412 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:56:31.412 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:31.412 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:31.413 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:31.413 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:31.413 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:31.414 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:31.414 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:31.414 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:31.414 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:31.415 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:31.415 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:56:31.415 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:31.415 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:56:31.416 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:56:31.416 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:56:31.416 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:31.416 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:56:31.418 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.875533_162.1 2021-09-02T07:56:36.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:36.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:36.421 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:56:36.477 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:56:36.478 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:56:36.778 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:56:36.779 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:56:36.779 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:56:36.779 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:56:36.780 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:56:36.780 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:56:36.780 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:36.780 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:36.780 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:56:36.781 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:36.781 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:56:36.781 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:36.781 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:36.782 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:56:36.782 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:36.782 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:56:36.782 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:36.783 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:56:36.783 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:36.783 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:36.783 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:36.784 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:36.784 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:56:36.784 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:36.784 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:36.784 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.099221_173.1", 2021-09-02T07:56:36.785 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:36.785 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:36.785 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:36.785 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.099221Z", 2021-09-02T07:56:36.786 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:36.786 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:36.787 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:36.787 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:36.787 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:56:36.787 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:36.788 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:36.788 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:36.788 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:36.788 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:36.789 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:36.789 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:36.789 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:36.789 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:36.789 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:36.790 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:36.790 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:56:36.790 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:36.790 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:36.791 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:36.791 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:36.791 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:36.791 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:36.792 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:36.792 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:36.792 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:36.792 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:36.792 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:56:36.793 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:36.793 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:56:36.793 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:56:36.793 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:56:36.794 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:36.794 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:56:36.794 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.099221_173.1 2021-09-02T07:56:41.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:41.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:41.797 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:56:41.883 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:56:41.883 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:56:42.125 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:56:42.125 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:56:42.125 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:56:42.126 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:56:42.126 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:56:42.126 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:56:42.126 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:42.127 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:42.127 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:56:42.127 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:42.127 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:56:42.127 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:42.128 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:42.128 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:56:42.128 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:42.128 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:56:42.129 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:42.129 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:56:42.129 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:42.129 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:42.130 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:42.130 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:42.130 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:56:42.130 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:42.131 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:42.131 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.168223_183.1", 2021-09-02T07:56:42.131 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:42.131 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:42.131 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:42.132 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.168223Z", 2021-09-02T07:56:42.132 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:42.132 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:42.132 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:42.133 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:42.133 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:56:42.133 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:42.133 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:42.133 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:42.134 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:42.134 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:42.135 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:42.135 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:42.135 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:42.135 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:42.136 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:42.136 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:42.136 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:56:42.136 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:42.137 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:42.137 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:42.137 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:42.137 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:42.138 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:42.138 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:42.138 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:42.138 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:42.139 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:42.139 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:56:42.139 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:42.139 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:56:42.140 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:56:42.140 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:56:42.140 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:42.140 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:56:42.142 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.168223_183.1 2021-09-02T07:56:47.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:47.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:47.144 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:56:47.187 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:56:47.188 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:56:47.272 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:56:47.272 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:56:47.272 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:56:47.273 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:56:47.273 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:56:47.273 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:56:47.273 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:47.274 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:47.274 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:56:47.274 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:47.274 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:56:47.274 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:47.275 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:47.275 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:56:47.275 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:47.275 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:56:47.276 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:47.276 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:56:47.276 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:47.276 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:47.277 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:47.277 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:47.277 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:56:47.278 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:47.278 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:47.278 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.911309_165.1", 2021-09-02T07:56:47.278 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:47.278 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:47.279 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:47.279 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.911309Z", 2021-09-02T07:56:47.279 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:47.279 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:47.280 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:47.280 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:47.280 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:56:47.280 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:47.281 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:47.281 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:47.281 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:47.281 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:47.282 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:47.282 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:47.282 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:47.282 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:47.282 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:47.283 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:47.283 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:56:47.283 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:47.283 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:47.284 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:47.284 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:47.284 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:47.284 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:47.285 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:47.285 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:47.285 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:47.285 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:47.286 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:56:47.286 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:47.286 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:56:47.286 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:56:47.286 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:56:47.287 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:47.287 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:56:47.288 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.911309_165.1 2021-09-02T07:56:52.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:52.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:52.292 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:56:52.340 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:56:52.340 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:56:52.433 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:56:52.433 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:56:52.433 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:56:52.434 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:56:52.434 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:56:52.434 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:56:52.434 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:52.434 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:52.435 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:56:52.435 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:52.435 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:56:52.435 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:52.436 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:52.436 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:56:52.436 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:52.437 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:56:52.438 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:52.439 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:56:52.439 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:52.439 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:52.439 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:52.440 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:52.440 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:56:52.440 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:52.440 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:52.441 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.107268_176.1", 2021-09-02T07:56:52.441 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:52.441 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:52.441 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:52.442 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.107268Z", 2021-09-02T07:56:52.442 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:52.442 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:52.442 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:52.443 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:52.443 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:56:52.443 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:52.443 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:52.444 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:52.444 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:52.444 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:52.444 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:52.444 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:52.445 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:52.445 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:52.445 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:52.445 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:52.446 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:56:52.446 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:52.446 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:52.446 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:52.447 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:52.447 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:52.447 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:52.447 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:52.448 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:52.448 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:52.448 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:52.448 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:56:52.448 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:52.449 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:56:52.449 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:56:52.449 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:56:52.449 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:52.450 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:56:52.451 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.107268_176.1 2021-09-02T07:56:57.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:57.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:56:57.454 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:56:57.498 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:56:57.498 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:56:57.578 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:56:57.579 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:56:57.579 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:56:57.579 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:56:57.580 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:56:57.580 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:56:57.580 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:57.580 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:57.580 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:56:57.581 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:57.581 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:56:57.581 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:57.581 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:57.582 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:56:57.582 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:57.586 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:56:57.587 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:57.589 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:56:57.589 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:57.589 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:57.590 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:57.590 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:57.591 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:56:57.591 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:57.592 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:57.592 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T07:56:57.593 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:57.593 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:57.593 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:57.594 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T07:56:57.594 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:57.595 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:57.595 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:57.596 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:57.596 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:56:57.597 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:57.597 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:57.597 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:57.598 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:57.598 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:57.599 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:57.600 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:57.600 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:57.601 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:57.601 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:57.601 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:56:57.601 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:56:57.602 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:56:57.602 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:56:57.602 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:56:57.603 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:56:57.603 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:56:57.603 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:56:57.603 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:56:57.604 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:56:57.604 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:57.604 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:57.604 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:56:57.605 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:56:57.605 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:56:57.605 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:56:57.606 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:56:57.606 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:56:57.606 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:56:57.607 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.906521_163.1 2021-09-02T07:57:02.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:02.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:02.610 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:57:02.678 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:57:02.678 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:57:03.082 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:57:03.082 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:57:03.083 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:57:03.083 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:57:03.083 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:57:03.083 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:57:03.084 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:03.084 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:03.084 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:57:03.084 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:03.084 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:57:03.085 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:03.085 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:03.085 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:57:03.085 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:03.086 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:57:03.086 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:03.086 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:57:03.086 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:03.087 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:03.087 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:03.087 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:03.087 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:57:03.087 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:03.088 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:03.088 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.932719_168.1", 2021-09-02T07:57:03.088 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:03.088 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:03.088 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:03.089 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.932719Z", 2021-09-02T07:57:03.089 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:03.089 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:03.089 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:03.090 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:03.090 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:57:03.090 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:03.090 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:03.091 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:03.091 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:03.091 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:03.091 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:03.091 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:03.092 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:03.092 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:03.092 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:03.092 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:03.093 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:57:03.093 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:03.093 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:03.094 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:03.094 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:03.094 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:03.094 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:03.094 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:03.095 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:03.095 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:03.095 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:03.095 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:57:03.096 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:03.096 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:57:03.096 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:57:03.096 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:57:03.097 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:03.098 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:57:03.099 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.932719_168.1 2021-09-02T07:57:08.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:08.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:08.102 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:57:08.160 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:57:08.160 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:57:08.277 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:57:08.277 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:57:08.277 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:57:08.278 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:57:08.278 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:57:08.278 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:57:08.278 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:08.278 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:08.279 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:57:08.279 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:08.279 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:57:08.279 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:08.280 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:08.280 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:57:08.280 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:08.280 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:57:08.281 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:08.281 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:57:08.281 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:08.281 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:08.282 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:08.282 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:08.282 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:57:08.282 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:08.282 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:08.283 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.932719_168.1", 2021-09-02T07:57:08.283 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:08.283 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:08.283 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:08.284 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.932719Z", 2021-09-02T07:57:08.284 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:08.284 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:08.285 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:08.285 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:08.285 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:57:08.285 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:08.286 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:08.286 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:08.286 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:08.286 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:08.286 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:08.287 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:08.287 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:08.287 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:08.287 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:08.288 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:08.288 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:57:08.288 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:08.289 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:08.289 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:08.289 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:08.289 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:08.290 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:08.290 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:08.290 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:08.290 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:08.291 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:08.291 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:57:08.291 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:08.291 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:57:08.292 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:57:08.292 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:57:08.292 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:08.292 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:57:08.294 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.932719_168.1 2021-09-02T07:57:13.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:13.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:13.297 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:57:13.391 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:57:13.392 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:57:13.844 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:57:13.844 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:57:13.845 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:57:13.845 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:57:13.845 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:57:13.845 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:57:13.846 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:13.846 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:13.847 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:57:13.847 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:13.847 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:57:13.848 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:13.848 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:13.848 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:57:13.848 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:13.848 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:57:13.849 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:13.849 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:57:13.849 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:13.850 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:13.850 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:13.851 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:13.851 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:57:13.852 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:13.852 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:13.852 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T07:57:13.852 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:13.853 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:13.853 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:13.853 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T07:57:13.853 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:13.854 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:13.854 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:13.855 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:13.855 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:57:13.855 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:13.855 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:13.856 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:13.856 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:13.856 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:13.856 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:13.857 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:13.857 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:13.857 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:13.857 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:13.858 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:13.858 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:57:13.858 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:13.859 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:13.859 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:13.859 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:13.860 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:13.860 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:13.860 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:13.861 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:13.861 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:13.861 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:13.861 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:57:13.861 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:13.862 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:57:13.862 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:57:13.862 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:57:13.862 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:13.863 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:57:13.864 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.906521_163.1 2021-09-02T07:57:18.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:18.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:18.867 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:57:18.954 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:57:18.955 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:57:19.502 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:57:19.502 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:57:19.503 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:57:19.503 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:57:19.503 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:57:19.503 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:57:19.504 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:19.504 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:19.504 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:57:19.504 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:19.504 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:57:19.505 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:19.505 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:19.505 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:57:19.505 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:19.506 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:57:19.506 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:19.506 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:57:19.506 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:19.507 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:19.507 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:19.507 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:19.507 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:57:19.507 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:19.508 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:19.508 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.039849_171.1", 2021-09-02T07:57:19.508 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:19.508 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:19.509 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:19.509 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.039849Z", 2021-09-02T07:57:19.509 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:19.509 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:19.510 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:19.510 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:19.511 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:57:19.511 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:19.511 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:19.511 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:19.512 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:19.512 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:19.512 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:19.512 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:19.513 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:19.513 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:19.513 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:19.513 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:19.514 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:57:19.514 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:19.514 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:19.514 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:19.515 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:19.515 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:19.515 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:19.515 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:19.516 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:19.516 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:19.516 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:19.516 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:57:19.516 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:19.517 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:57:19.517 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:57:19.517 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:57:19.517 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:19.518 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:57:19.519 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.039849_171.1 2021-09-02T07:57:24.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:24.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:24.522 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:57:24.570 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:57:24.570 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:57:24.659 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:57:24.660 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:57:24.660 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:57:24.660 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:57:24.661 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:57:24.661 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:57:24.661 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:24.661 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:24.662 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:57:24.662 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:24.662 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:57:24.662 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:24.663 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:24.663 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:57:24.663 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:24.663 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:57:24.663 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:24.664 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:57:24.664 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:24.664 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:24.664 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:24.664 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:24.665 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:57:24.665 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:24.665 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:24.666 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.129118_179.1", 2021-09-02T07:57:24.666 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:24.666 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:24.666 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:24.666 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.129118Z", 2021-09-02T07:57:24.667 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:24.667 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:24.667 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:24.667 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:24.668 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:57:24.668 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:24.668 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:24.668 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:24.668 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:24.669 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:24.669 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:24.669 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:24.669 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:24.670 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:24.670 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:24.670 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:24.671 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:57:24.671 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:24.671 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:24.671 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:24.671 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:24.672 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:24.672 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:24.672 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:24.672 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:24.673 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:24.673 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:24.673 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:57:24.673 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:24.673 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:57:24.674 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:57:24.674 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:57:24.674 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:24.674 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:57:24.676 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.129118_179.1 2021-09-02T07:57:29.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:29.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:29.679 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:57:29.727 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:57:29.728 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:57:29.797 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:57:29.798 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:57:29.798 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:57:29.798 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:57:29.798 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:57:29.799 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:57:29.799 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:29.799 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:29.799 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:57:29.800 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:29.800 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:57:29.800 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:29.800 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:29.801 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:57:29.801 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:29.801 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:57:29.801 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:29.801 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:57:29.802 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:29.802 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:29.802 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:29.802 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:29.803 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:57:29.803 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:29.803 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:29.803 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.102736_174.1", 2021-09-02T07:57:29.804 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:29.804 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:29.804 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:29.804 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.102736Z", 2021-09-02T07:57:29.805 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:29.805 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:29.805 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:29.805 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:29.805 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:57:29.806 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:29.806 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:29.806 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:29.806 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:29.807 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:29.807 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:29.807 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:29.807 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:29.808 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:29.808 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:29.808 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:29.808 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:57:29.808 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:29.809 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:29.809 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:29.809 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:29.809 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:29.810 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:29.810 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:29.810 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:29.810 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:29.811 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:29.811 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:57:29.811 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:29.811 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:57:29.812 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:57:29.812 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:57:29.812 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:29.812 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:57:29.814 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.102736_174.1 2021-09-02T07:57:34.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:34.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:34.816 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:57:34.861 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:57:34.861 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:57:34.944 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:57:34.945 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:57:34.945 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:57:34.945 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:57:34.946 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:57:34.946 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:57:34.946 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:34.946 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:34.947 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:57:34.947 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:34.947 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:57:34.947 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:34.948 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:34.948 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:57:34.949 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:34.949 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:57:34.949 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:34.950 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:57:34.952 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:34.953 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:34.953 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:34.954 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:34.954 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:57:34.954 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:34.955 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:34.955 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.929348_167.1", 2021-09-02T07:57:34.955 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:34.955 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:34.956 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:34.956 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.929348Z", 2021-09-02T07:57:34.956 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:34.956 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:34.957 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:34.957 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:34.957 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:57:34.957 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:34.958 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:34.958 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:34.958 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:34.958 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:34.959 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:34.959 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:34.959 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:34.959 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:34.960 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:34.960 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:34.961 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:57:34.962 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:34.962 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:34.962 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:34.963 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:34.963 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:34.963 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:34.963 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:34.964 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:34.964 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:34.964 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:34.964 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:57:34.965 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:34.965 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:57:34.965 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:57:34.965 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:57:34.966 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:34.966 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:57:34.966 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.929348_167.1 2021-09-02T07:57:39.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:39.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:39.969 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:57:40.050 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:57:40.050 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:57:40.365 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:57:40.366 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:57:40.366 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:57:40.366 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:57:40.366 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:57:40.367 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:57:40.367 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:40.367 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:40.367 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:57:40.368 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:40.368 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:57:40.369 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:40.370 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:40.370 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:57:40.370 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:40.370 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:57:40.371 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:40.371 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:57:40.371 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:40.371 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:40.372 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:40.372 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:40.372 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:57:40.372 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:40.373 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:40.373 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T07:57:40.373 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:40.373 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:40.374 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:40.374 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T07:57:40.374 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:40.375 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:40.375 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:40.375 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:40.375 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:57:40.376 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:40.376 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:40.376 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:40.376 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:40.376 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:40.377 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:40.377 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:40.377 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:40.377 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:40.378 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:40.378 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:40.378 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:57:40.379 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:40.379 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:40.379 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:40.379 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:40.380 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:40.380 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:40.380 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:40.380 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:40.381 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:40.381 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:40.381 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:57:40.381 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:40.382 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:57:40.382 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:57:40.382 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:57:40.382 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:40.383 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:57:40.384 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.910082_164.1 2021-09-02T07:57:45.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:45.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:45.387 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:57:45.429 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:57:45.430 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:57:45.562 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:57:45.562 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:57:45.562 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:57:45.563 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:57:45.563 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:57:45.563 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:57:45.563 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:45.564 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:45.564 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:57:45.564 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:45.564 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:57:45.565 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:45.565 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:45.565 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:57:45.565 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:45.565 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:57:45.566 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:45.567 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:57:45.567 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:45.567 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:45.567 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:45.567 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:45.568 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:57:45.568 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:45.568 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:45.569 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.911309_165.1", 2021-09-02T07:57:45.569 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:45.569 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:45.569 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:45.570 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.911309Z", 2021-09-02T07:57:45.570 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:45.570 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:45.570 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:45.571 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:45.571 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:57:45.571 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:45.571 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:45.572 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:45.572 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:45.572 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:45.573 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:45.573 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:45.573 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:45.574 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:45.574 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:45.574 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:45.574 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:57:45.575 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:45.575 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:45.575 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:45.575 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:45.576 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:45.576 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:45.576 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:45.576 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:45.577 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:45.577 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:45.577 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:57:45.577 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:45.578 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:57:45.578 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:57:45.578 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:57:45.580 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:45.583 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:57:45.584 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.911309_165.1 2021-09-02T07:57:50.586 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:50.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:50.590 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:57:50.684 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:57:50.685 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:57:50.927 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:57:50.928 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:57:50.930 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:57:50.931 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:57:50.932 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:57:50.932 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:57:50.932 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:50.933 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:50.933 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:57:50.934 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:50.934 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:57:50.935 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:50.935 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:50.936 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:57:50.936 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:50.936 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:57:50.937 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:50.937 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:57:50.938 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:50.938 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:50.939 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:50.939 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:50.940 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:57:50.940 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:50.940 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:50.941 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.911309_165.1", 2021-09-02T07:57:50.941 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:50.942 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:50.942 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:50.943 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.911309Z", 2021-09-02T07:57:50.943 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:50.944 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:50.944 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:50.944 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:50.945 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:57:50.945 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:50.946 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:50.946 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:50.947 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:50.947 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:50.948 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:50.948 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:50.948 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:50.949 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:50.949 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:50.950 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:50.950 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:57:50.951 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:50.951 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:50.951 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:50.952 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:50.952 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:50.953 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:50.953 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:50.954 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:50.954 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:50.955 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:50.955 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:57:50.956 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:50.956 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:57:50.956 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:57:50.957 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:57:50.957 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:50.958 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:57:50.960 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.911309_165.1 2021-09-02T07:57:55.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:55.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:57:55.963 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:57:56.066 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:57:56.066 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:57:56.515 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:57:56.515 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:57:56.516 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:57:56.516 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:57:56.516 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:57:56.516 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:57:56.516 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:56.517 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:56.517 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:57:56.517 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:56.517 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:57:56.518 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:56.518 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:56.518 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:57:56.519 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:56.519 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:57:56.519 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:56.520 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:57:56.520 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:56.520 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:56.520 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:56.521 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:56.521 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:57:56.521 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:56.521 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:56.521 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.911309_165.1", 2021-09-02T07:57:56.522 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:56.522 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:56.522 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:56.522 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.911309Z", 2021-09-02T07:57:56.523 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:56.523 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:56.523 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:56.523 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:56.524 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:57:56.524 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:56.524 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:56.524 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:56.525 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:56.525 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:56.525 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:56.525 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:56.525 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:56.526 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:56.526 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:56.526 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:57:56.526 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:57:56.527 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:57:56.527 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:57:56.527 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:57:56.527 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:57:56.528 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:57:56.528 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:57:56.528 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:57:56.528 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:57:56.529 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:56.529 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:56.529 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:57:56.529 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:57:56.529 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:57:56.530 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:57:56.530 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:57:56.530 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:57:56.531 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:57:56.532 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.911309_165.1 2021-09-02T07:58:01.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:01.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:01.536 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:58:01.614 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:58:01.615 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:58:01.748 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:58:01.749 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:58:01.749 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:58:01.749 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:58:01.750 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:58:01.750 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:58:01.750 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:01.750 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:01.751 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:58:01.751 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:01.752 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:58:01.752 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:01.752 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:01.752 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:58:01.753 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:01.753 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:58:01.754 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:01.754 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:58:01.755 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:01.756 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:01.757 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:01.757 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:01.759 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:58:01.760 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:01.760 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:01.761 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.102736_174.1", 2021-09-02T07:58:01.763 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:01.765 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:01.765 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:01.766 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.102736Z", 2021-09-02T07:58:01.766 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:01.766 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:01.767 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:01.767 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:01.767 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:58:01.768 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:01.768 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:01.768 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:01.768 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:01.768 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:01.769 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:01.769 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:01.769 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:01.769 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:01.770 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:01.770 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:01.770 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:58:01.771 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:01.771 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:01.771 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:01.772 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:01.772 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:01.772 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:01.772 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:01.772 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:01.773 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:01.773 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:01.773 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:58:01.773 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:01.774 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:58:01.774 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:58:01.774 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:58:01.774 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:01.774 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:58:01.775 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.102736_174.1 2021-09-02T07:58:06.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:06.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:06.777 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:58:06.826 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:58:06.826 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:58:06.900 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:58:06.901 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:58:06.901 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:58:06.902 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:58:06.902 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:58:06.902 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:58:06.902 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:06.903 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:06.903 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:58:06.903 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:06.904 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:58:06.904 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:06.904 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:06.904 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:58:06.905 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:06.905 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:58:06.905 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:06.905 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:58:06.905 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:06.906 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:06.906 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:06.906 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:06.906 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:58:06.907 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:06.907 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:06.907 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T07:58:06.907 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:06.908 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:06.908 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:06.908 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T07:58:06.908 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:06.908 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:06.909 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:06.909 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:06.909 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:58:06.909 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:06.910 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:06.910 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:06.910 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:06.910 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:06.911 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:06.911 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:06.911 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:06.911 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:06.912 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:06.912 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:06.912 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:58:06.912 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:06.912 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:06.913 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:06.913 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:06.913 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:06.913 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:06.914 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:06.914 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:06.914 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:06.914 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:06.914 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:58:06.915 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:06.915 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:58:06.915 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:58:06.916 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:58:06.916 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:06.916 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:58:06.918 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.875533_162.1 2021-09-02T07:58:11.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:11.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:11.920 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:58:11.962 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:58:11.962 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:58:12.028 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:58:12.029 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:58:12.030 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:58:12.030 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:58:12.031 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:58:12.031 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:58:12.031 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:12.032 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:12.032 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:58:12.033 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:12.033 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:58:12.034 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:12.035 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:12.035 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:58:12.036 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:12.036 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:58:12.036 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:12.036 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:58:12.036 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:12.037 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:12.037 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:12.037 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:12.037 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:58:12.038 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:12.038 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:12.038 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.193730_185.1", 2021-09-02T07:58:12.038 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:12.039 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:12.039 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:12.039 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.193730Z", 2021-09-02T07:58:12.039 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:12.039 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:12.040 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:12.040 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:12.040 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:58:12.040 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:12.041 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:12.041 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:12.041 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:12.041 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:12.042 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:12.042 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:12.042 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:12.042 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:12.042 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:12.043 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:12.043 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:58:12.043 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:12.043 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:12.044 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:12.044 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:12.044 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:12.044 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:12.044 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:12.045 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:12.045 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:12.045 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:12.045 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:58:12.046 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:12.046 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:58:12.046 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:58:12.046 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:58:12.047 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:12.047 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:58:12.048 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.193730_185.1 2021-09-02T07:58:17.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:17.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:17.052 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:58:17.140 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:58:17.141 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:58:17.365 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:58:17.365 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:58:17.365 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:58:17.366 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:58:17.366 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:58:17.366 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:58:17.366 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:17.367 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:17.367 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:58:17.367 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:17.367 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:58:17.367 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:17.368 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:17.368 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:58:17.368 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:17.368 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:58:17.369 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:17.369 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:58:17.369 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:17.369 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:17.370 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:17.370 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:17.370 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:58:17.371 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:17.371 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:17.371 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.985199_169.1", 2021-09-02T07:58:17.371 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:17.372 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:17.372 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:17.372 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.985199Z", 2021-09-02T07:58:17.372 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:17.372 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:17.373 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:17.373 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:17.373 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:58:17.373 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:17.374 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:17.374 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:17.374 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:17.375 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:17.375 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:17.375 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:17.375 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:17.376 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:17.376 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:17.376 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:17.376 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:58:17.377 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:17.377 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:17.377 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:17.377 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:17.377 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:17.378 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:17.378 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:17.378 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:17.378 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:17.379 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:17.379 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:58:17.379 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:17.380 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:58:17.380 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:58:17.380 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:58:17.380 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:17.381 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:58:17.382 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.985199_169.1 2021-09-02T07:58:22.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:22.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:22.385 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:58:22.464 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:58:22.464 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:58:22.605 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:58:22.605 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:58:22.606 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:58:22.606 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:58:22.606 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:58:22.606 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:58:22.607 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:22.607 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:22.607 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:58:22.607 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:22.608 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:58:22.608 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:22.609 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:22.609 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:58:22.609 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:22.610 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:58:22.610 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:22.610 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:58:22.610 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:22.611 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:22.611 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:22.611 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:22.612 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:58:22.612 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:22.612 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:22.612 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T07:58:22.613 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:22.613 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:22.613 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:22.613 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T07:58:22.614 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:22.614 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:22.615 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:22.615 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:22.615 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:58:22.615 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:22.616 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:22.616 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:22.616 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:22.617 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:22.617 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:22.617 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:22.617 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:22.618 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:22.618 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:22.618 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:22.618 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:58:22.618 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:22.619 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:22.619 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:22.619 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:22.619 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:22.620 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:22.620 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:22.621 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:22.621 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:22.622 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:22.622 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:58:22.622 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:22.622 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:58:22.623 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:58:22.624 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:58:22.624 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:22.625 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:58:22.625 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.910082_164.1 2021-09-02T07:58:27.627 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:27.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:27.628 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:58:27.727 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:58:27.727 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:58:28.219 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:58:28.219 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:58:28.220 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:58:28.220 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:58:28.220 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:58:28.220 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:58:28.221 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:28.221 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:28.221 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:58:28.222 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:28.222 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:58:28.222 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:28.223 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:28.223 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:58:28.224 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:28.224 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:58:28.224 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:28.225 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:58:28.225 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:28.226 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:28.226 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:28.227 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:28.227 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:58:28.227 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:28.228 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:28.228 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T07:58:28.229 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:28.229 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:28.230 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:28.230 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T07:58:28.230 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:28.231 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:28.231 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:28.232 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:28.232 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:58:28.232 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:28.233 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:28.233 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:28.233 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:28.234 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:28.234 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:28.234 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:28.234 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:28.235 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:28.235 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:28.235 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:28.235 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:58:28.235 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:28.236 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:28.236 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:28.236 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:28.236 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:28.237 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:28.237 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:28.237 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:28.237 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:28.237 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:28.238 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:58:28.238 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:28.238 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:58:28.238 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:58:28.239 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:58:28.239 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:28.239 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:58:28.241 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.906521_163.1 2021-09-02T07:58:33.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:33.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:33.244 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:58:33.340 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:58:33.340 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:58:33.807 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:58:33.807 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:58:33.808 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:58:33.808 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:58:33.808 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:58:33.808 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:58:33.809 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:33.809 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:33.809 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:58:33.809 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:33.810 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:58:33.810 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:33.810 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:33.810 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:58:33.811 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:33.811 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:58:33.811 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:33.811 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:58:33.811 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:33.812 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:33.812 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:33.812 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:33.812 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:58:33.813 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:33.813 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:33.813 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.932719_168.1", 2021-09-02T07:58:33.813 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:33.813 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:33.814 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:33.814 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.932719Z", 2021-09-02T07:58:33.814 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:33.814 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:33.815 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:33.815 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:33.815 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:58:33.815 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:33.815 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:33.816 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:33.816 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:33.816 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:33.816 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:33.817 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:33.817 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:33.817 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:33.817 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:33.817 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:33.818 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:58:33.818 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:33.818 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:33.818 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:33.819 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:33.819 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:33.819 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:33.819 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:33.820 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:33.820 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:33.820 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:33.820 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:58:33.820 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:33.821 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:58:33.821 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:58:33.821 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:58:33.821 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:33.822 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:58:33.823 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.932719_168.1 2021-09-02T07:58:38.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:38.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:38.828 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:58:38.872 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:58:38.873 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:58:38.947 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:58:38.947 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:58:38.948 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:58:38.948 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:58:38.948 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:58:38.949 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:58:38.949 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:38.949 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:38.950 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:58:38.950 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:38.950 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:58:38.950 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:38.951 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:38.951 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:58:38.951 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:38.951 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:58:38.952 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:38.952 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:58:38.952 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:38.952 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:38.953 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:38.953 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:38.953 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:58:38.953 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:38.954 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:38.954 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T07:58:38.954 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:38.954 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:38.955 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:38.955 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T07:58:38.955 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:38.955 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:38.956 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:38.956 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:38.956 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:58:38.957 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:38.957 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:38.957 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:38.957 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:38.958 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:38.958 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:38.958 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:38.958 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:38.959 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:38.959 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:38.959 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:38.959 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:58:38.960 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:38.960 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:38.960 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:38.960 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:38.961 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:38.961 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:38.961 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:38.961 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:38.962 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:38.962 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:38.962 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:58:38.962 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:38.963 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:58:38.963 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:58:38.963 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:58:38.963 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:38.964 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:58:38.965 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.875533_162.1 2021-09-02T07:58:43.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:43.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:43.968 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:58:44.017 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:58:44.017 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:58:44.105 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:58:44.106 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:58:44.106 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:58:44.107 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:58:44.107 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:58:44.107 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:58:44.108 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:44.108 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:44.109 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:58:44.110 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:44.110 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:58:44.111 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:44.111 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:44.111 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:58:44.112 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:44.112 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:58:44.113 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:44.113 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:58:44.114 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:44.114 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:44.115 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:44.115 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:44.115 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:58:44.116 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:44.116 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:44.117 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.043408_172.1", 2021-09-02T07:58:44.117 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:44.118 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:44.118 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:44.119 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.043408Z", 2021-09-02T07:58:44.119 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:44.119 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:44.120 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:44.121 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:44.121 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:58:44.122 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:44.122 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:44.123 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:44.123 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:44.123 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:44.124 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:44.124 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:44.124 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:44.125 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:44.125 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:44.125 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:44.126 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:58:44.126 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:44.126 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:44.127 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:44.127 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:44.127 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:44.128 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:44.128 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:44.128 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:44.128 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:44.128 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:44.129 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:58:44.129 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:44.129 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:58:44.130 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:58:44.130 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:58:44.130 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:44.131 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:58:44.132 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.043408_172.1 2021-09-02T07:58:49.134 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:49.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:49.135 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:58:49.234 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:58:49.234 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:58:49.438 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:58:49.439 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:58:49.439 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:58:49.439 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:58:49.440 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:58:49.440 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:58:49.440 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:49.440 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:49.440 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:58:49.441 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:49.441 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:58:49.441 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:49.441 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:49.442 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:58:49.442 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:49.442 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:58:49.442 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:49.443 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:58:49.443 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:49.443 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:49.443 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:49.443 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:49.444 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:58:49.444 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:49.444 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:49.444 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T07:58:49.445 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:49.445 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:49.445 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:49.446 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T07:58:49.446 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:49.446 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:49.446 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:49.447 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:49.447 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:58:49.447 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:49.447 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:49.447 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:49.448 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:49.448 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:49.448 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:49.448 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:49.449 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:49.449 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:49.449 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:49.449 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:49.450 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:58:49.450 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:49.450 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:49.450 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:49.451 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:49.451 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:49.451 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:49.451 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:49.451 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:49.452 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:49.452 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:49.452 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:58:49.453 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:49.453 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:58:49.453 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:58:49.454 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:58:49.454 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:49.454 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:58:49.455 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.103991_175.1 2021-09-02T07:58:54.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:54.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:58:54.458 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:58:54.522 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:58:54.522 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:58:55.030 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:58:55.031 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:58:55.031 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:58:55.032 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:58:55.032 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:58:55.033 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:58:55.033 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:55.034 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:55.034 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:58:55.035 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:55.035 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:58:55.037 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:55.037 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:55.037 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:58:55.038 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:55.038 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:58:55.039 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:55.039 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:58:55.040 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:55.040 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:55.041 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:55.041 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:55.041 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:58:55.042 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:55.042 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:55.043 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T07:58:55.043 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:55.044 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:55.044 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:55.044 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T07:58:55.045 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:55.045 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:55.046 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:55.046 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:55.047 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:58:55.048 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:55.048 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:55.049 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:55.049 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:55.050 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:55.050 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:55.050 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:55.052 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:55.052 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:55.052 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:55.053 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:58:55.053 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:58:55.053 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:58:55.053 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:58:55.054 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:58:55.054 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:58:55.054 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:58:55.054 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:58:55.055 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:58:55.055 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:58:55.055 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:55.055 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:55.056 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:58:55.056 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:58:55.056 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:58:55.056 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:58:55.057 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:58:55.057 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:58:55.057 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:58:55.058 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.875533_162.1 2021-09-02T07:59:00.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:00.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:00.061 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:59:00.103 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:59:00.103 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:59:00.182 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:59:00.182 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:59:00.183 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:59:00.183 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:59:00.183 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:59:00.183 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:59:00.184 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:00.184 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:00.184 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:59:00.184 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:00.185 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:59:00.185 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:00.185 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:00.185 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:59:00.186 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:00.186 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:59:00.186 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:00.187 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:59:00.187 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:00.187 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:00.187 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:00.188 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:00.188 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:59:00.188 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:00.188 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:00.189 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T07:59:00.189 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:00.189 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:00.189 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:00.189 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T07:59:00.190 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:00.190 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:00.190 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:00.191 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:00.191 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:59:00.191 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:00.191 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:00.191 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:00.192 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:00.192 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:00.192 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:00.192 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:00.193 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:00.193 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:00.193 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:00.193 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:00.194 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:59:00.194 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:00.194 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:00.194 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:00.195 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:00.195 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:00.195 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:00.195 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:00.196 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:00.196 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:00.196 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:00.196 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:59:00.197 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:00.197 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:59:00.197 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:59:00.197 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:59:00.197 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:00.198 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:59:00.199 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.910082_164.1 2021-09-02T07:59:05.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:05.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:05.202 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:59:05.288 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:59:05.288 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:59:05.841 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:59:05.841 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:59:05.841 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:59:05.841 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:59:05.842 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:59:05.842 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:59:05.842 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:05.842 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:05.843 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:59:05.843 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:05.843 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:59:05.843 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:05.844 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:05.844 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:59:05.844 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:05.844 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:59:05.844 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:05.845 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:59:05.845 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:05.845 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:05.845 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:05.846 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:05.846 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:59:05.846 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:05.846 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:05.846 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T07:59:05.847 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:05.847 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:05.847 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:05.847 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T07:59:05.848 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:05.848 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:05.848 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:05.848 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:05.849 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:59:05.849 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:05.849 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:05.849 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:05.849 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:05.850 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:05.850 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:05.850 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:05.850 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:05.851 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:05.851 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:05.851 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:05.851 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:59:05.851 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:05.852 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:05.852 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:05.852 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:05.852 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:05.853 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:05.853 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:05.853 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:05.853 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:05.853 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:05.854 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:59:05.854 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:05.854 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:59:05.854 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:59:05.854 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:59:05.855 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:05.855 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:59:05.856 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.910082_164.1 2021-09-02T07:59:10.858 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:10.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:10.859 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:59:10.927 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:59:10.927 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:59:11.533 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:59:11.533 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:59:11.533 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:59:11.533 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:59:11.534 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:59:11.534 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:59:11.534 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:11.534 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:11.535 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:59:11.535 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:11.535 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:59:11.535 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:11.536 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:11.536 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:59:11.536 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:11.536 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:59:11.539 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:11.539 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:59:11.539 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:11.540 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:11.540 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:11.540 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:11.540 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:59:11.541 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:11.541 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:11.541 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T07:59:11.541 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:11.541 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:11.542 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:11.542 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T07:59:11.542 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:11.542 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:11.543 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:11.543 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:11.543 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:59:11.543 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:11.543 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:11.544 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:11.544 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:11.544 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:11.544 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:11.545 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:11.545 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:11.545 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:11.545 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:11.545 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:11.546 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:59:11.546 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:11.546 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:11.546 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:11.547 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:11.547 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:11.547 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:11.547 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:11.548 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:11.548 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:11.548 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:11.548 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:59:11.548 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:11.549 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:59:11.549 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:59:11.549 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:59:11.549 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:11.549 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:59:11.551 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.103991_175.1 2021-09-02T07:59:16.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:16.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:16.554 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:59:16.601 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:59:16.602 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:59:16.748 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:59:16.748 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:59:16.749 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:59:16.749 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:59:16.749 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:59:16.749 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:59:16.750 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:16.750 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:16.750 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:59:16.750 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:16.751 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:59:16.751 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:16.751 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:16.751 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:59:16.752 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:16.752 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:59:16.752 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:16.752 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:59:16.753 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:16.753 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:16.753 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:16.753 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:16.753 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:59:16.754 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:16.754 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:16.754 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T07:59:16.754 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:16.755 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:16.755 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:16.755 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T07:59:16.755 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:16.756 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:16.756 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:16.756 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:16.756 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:59:16.757 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:16.757 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:16.757 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:16.757 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:16.757 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:16.758 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:16.758 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:16.758 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:16.758 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:16.759 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:16.759 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:16.759 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:59:16.759 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:16.760 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:16.760 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:16.760 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:16.760 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:16.760 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:16.761 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:16.761 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:16.761 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:16.761 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:16.762 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:59:16.762 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:16.762 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:59:16.762 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:59:16.763 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:59:16.763 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:16.763 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:59:16.765 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.103991_175.1 2021-09-02T07:59:21.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:21.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:21.767 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:59:21.815 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:59:21.816 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:59:21.890 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:59:21.890 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:59:21.891 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:59:21.891 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:59:21.891 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:59:21.891 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:59:21.891 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:21.892 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:21.892 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:59:21.892 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:21.892 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:59:21.893 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:21.893 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:21.893 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:59:21.893 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:21.894 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:59:21.894 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:21.894 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:59:21.894 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:21.894 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:21.895 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:21.895 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:21.895 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:59:21.895 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:21.896 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:21.896 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.911309_165.1", 2021-09-02T07:59:21.896 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:21.896 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:21.897 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:21.897 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.911309Z", 2021-09-02T07:59:21.897 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:21.897 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:21.897 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:21.898 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:21.898 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:59:21.898 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:21.898 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:21.899 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:21.899 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:21.899 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:21.899 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:21.899 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:21.900 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:21.906 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:21.906 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:21.907 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:21.907 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:59:21.907 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:21.907 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:21.908 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:21.908 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:21.908 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:21.908 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:21.908 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:21.909 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:21.909 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:21.909 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:21.909 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:59:21.910 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:21.910 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:59:21.910 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:59:21.910 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:59:21.911 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:21.911 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:59:21.912 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.911309_165.1 2021-09-02T07:59:26.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:26.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:26.915 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:59:26.966 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:59:26.966 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:59:27.045 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:59:27.046 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:59:27.046 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:59:27.046 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:59:27.046 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:59:27.047 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:59:27.047 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:27.047 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:27.047 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:59:27.047 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:27.048 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:59:27.048 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:27.048 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:27.048 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:59:27.049 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:27.049 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:59:27.049 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:27.049 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:59:27.049 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:27.050 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:27.050 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:27.050 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:27.050 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:59:27.051 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:27.051 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:27.051 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.990121_170.1", 2021-09-02T07:59:27.052 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:27.052 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:27.052 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:27.052 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.990121Z", 2021-09-02T07:59:27.052 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:27.053 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:27.053 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:27.054 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:27.054 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:59:27.054 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:27.055 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:27.056 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:27.056 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:27.056 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:27.057 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:27.057 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:27.059 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:27.060 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:27.060 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:27.060 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:27.060 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:59:27.061 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:27.061 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:27.061 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:27.061 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:27.062 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:27.062 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:27.062 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:27.062 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:27.063 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:27.063 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:27.063 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:59:27.063 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:27.063 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:59:27.064 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:59:27.064 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:59:27.064 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:27.064 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:59:27.066 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.990121_170.1 2021-09-02T07:59:32.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:32.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:32.070 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:59:32.127 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:59:32.128 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:59:32.430 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:59:32.430 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:59:32.431 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:59:32.431 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:59:32.432 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:59:32.432 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:59:32.433 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:32.433 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:32.433 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:59:32.433 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:32.434 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:59:32.434 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:32.434 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:32.434 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:59:32.435 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:32.435 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:59:32.435 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:32.435 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:59:32.435 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:32.436 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:32.436 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:32.436 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:32.436 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:59:32.437 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:32.437 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:32.437 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.168223_183.1", 2021-09-02T07:59:32.437 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:32.438 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:32.438 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:32.438 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.168223Z", 2021-09-02T07:59:32.438 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:32.439 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:32.439 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:32.439 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:32.439 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:59:32.440 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:32.440 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:32.440 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:32.440 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:32.441 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:32.441 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:32.441 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:32.441 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:32.441 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:32.442 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:32.442 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:32.442 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:59:32.442 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:32.443 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:32.443 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:32.443 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:32.443 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:32.444 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:32.444 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:32.444 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:32.444 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:32.445 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:32.445 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:59:32.445 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:32.445 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:59:32.445 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:59:32.446 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:59:32.446 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:32.446 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:59:32.448 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.168223_183.1 2021-09-02T07:59:37.450 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:37.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:37.452 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:59:37.536 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:59:37.537 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:59:37.763 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:59:37.765 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:59:37.765 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:59:37.765 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:59:37.766 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:59:37.766 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:59:37.767 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:37.767 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:37.767 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:59:37.767 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:37.768 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:59:37.768 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:37.768 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:37.769 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:59:37.769 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:37.769 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:59:37.769 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:37.770 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:59:37.770 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:37.770 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:37.770 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:37.771 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:37.772 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:59:37.772 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:37.772 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:37.773 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T07:59:37.773 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:37.773 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:37.773 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:37.774 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T07:59:37.774 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:37.775 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:37.775 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:37.776 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:37.776 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:59:37.776 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:37.776 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:37.777 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:37.777 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:37.777 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:37.777 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:37.778 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:37.778 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:37.778 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:37.778 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:37.779 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:37.779 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:59:37.779 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:37.779 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:37.780 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:37.780 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:37.780 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:37.780 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:37.780 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:37.781 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:37.781 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:37.781 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:37.781 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:59:37.782 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:37.782 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:59:37.782 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:59:37.782 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:59:37.783 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:37.783 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:59:37.783 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.103991_175.1 2021-09-02T07:59:42.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:42.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:42.787 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:59:42.831 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:59:42.832 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:59:42.903 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:59:42.903 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:59:42.903 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:59:42.904 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:59:42.904 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:59:42.904 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:59:42.904 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:42.905 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:42.905 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:59:42.905 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:42.905 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:59:42.905 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:42.906 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:42.908 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:59:42.908 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:42.908 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:59:42.909 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:42.909 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:59:42.909 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:42.909 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:42.910 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:42.910 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:42.911 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:59:42.911 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:42.911 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:42.912 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T07:59:42.912 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:42.913 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:42.913 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:42.913 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T07:59:42.913 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:42.914 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:42.914 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:42.914 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:42.915 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:59:42.915 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:42.915 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:42.915 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:42.916 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:42.916 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:42.916 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:42.916 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:42.916 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:42.917 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:42.917 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:42.917 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:42.917 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:59:42.918 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:42.918 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:42.919 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:42.919 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:42.919 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:42.919 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:42.920 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:42.920 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:42.920 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:42.920 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:42.920 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:59:42.921 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:42.921 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:59:42.921 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:59:42.921 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:59:42.922 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:42.922 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:59:42.923 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.875533_162.1 2021-09-02T07:59:47.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:47.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:47.926 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:59:47.972 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:59:47.972 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:59:48.050 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:59:48.051 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:59:48.051 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:59:48.051 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:59:48.052 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:59:48.052 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:59:48.052 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:48.052 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:48.053 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:59:48.053 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:48.053 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:59:48.053 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:48.054 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:48.054 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:59:48.054 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:48.054 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:59:48.055 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:48.055 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:59:48.055 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:48.055 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:48.056 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:48.056 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:48.056 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:59:48.056 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:48.057 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:48.057 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T07:59:48.057 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:48.057 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:48.057 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:48.058 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T07:59:48.058 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:48.058 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:48.058 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:48.059 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:48.059 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:59:48.059 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:48.059 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:48.060 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:48.060 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:48.060 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:48.060 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:48.060 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:48.061 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:48.061 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:48.061 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:48.061 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:48.062 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:59:48.062 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:48.062 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:48.062 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:48.062 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:48.063 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:48.063 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:48.063 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:48.063 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:48.064 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:48.064 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:48.064 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:59:48.064 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:48.064 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:59:48.065 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:59:48.065 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:59:48.065 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:48.065 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:59:48.066 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.875533_162.1 2021-09-02T07:59:53.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:53.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:53.068 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:59:53.139 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:59:53.139 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:59:53.269 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:59:53.269 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:59:53.269 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:59:53.270 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:59:53.270 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:59:53.270 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:59:53.270 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:53.271 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:53.271 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:59:53.271 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:53.271 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:59:53.272 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:53.272 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:53.272 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:59:53.272 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:53.273 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:59:53.273 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:53.273 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:59:53.273 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:53.273 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:53.274 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:53.274 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:53.274 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:59:53.274 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:53.275 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:53.275 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T07:59:53.275 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:53.275 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:53.276 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:53.276 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T07:59:53.276 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:53.276 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:53.276 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:53.277 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:53.277 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:59:53.277 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:53.278 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:53.278 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:53.278 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:53.279 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:53.279 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:53.279 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:53.279 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:53.280 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:53.280 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:53.280 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:53.280 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:59:53.281 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:53.281 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:53.281 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:53.281 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:53.282 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:53.282 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:53.282 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:53.282 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:53.283 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:53.283 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:53.283 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:59:53.283 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:53.283 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:59:53.284 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:59:53.284 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:59:53.284 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:53.284 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:59:53.286 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.910082_164.1 2021-09-02T07:59:58.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:58.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T07:59:58.289 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T07:59:58.358 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T07:59:58.358 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T07:59:58.469 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T07:59:58.470 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T07:59:58.470 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T07:59:58.471 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T07:59:58.471 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T07:59:58.471 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T07:59:58.471 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:58.471 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:58.472 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T07:59:58.472 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:58.472 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T07:59:58.472 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:58.473 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:58.473 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T07:59:58.473 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:58.473 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T07:59:58.473 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:58.474 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T07:59:58.474 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:58.474 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:58.474 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:58.475 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:58.475 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T07:59:58.475 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:58.475 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:58.475 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.914475_166.1", 2021-09-02T07:59:58.476 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:58.476 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:58.476 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:58.477 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.914475Z", 2021-09-02T07:59:58.477 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:58.477 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:58.477 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:58.477 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:58.478 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T07:59:58.478 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:58.478 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:58.478 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:58.479 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:58.479 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:58.480 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:58.480 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:58.480 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:58.480 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:58.481 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:58.481 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T07:59:58.481 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T07:59:58.481 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T07:59:58.483 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T07:59:58.483 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T07:59:58.483 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T07:59:58.483 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T07:59:58.484 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T07:59:58.484 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T07:59:58.484 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T07:59:58.484 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:58.484 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:58.485 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T07:59:58.485 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T07:59:58.485 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T07:59:58.486 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T07:59:58.486 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T07:59:58.486 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T07:59:58.486 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T07:59:58.488 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.914475_166.1 2021-09-02T08:00:03.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:03.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:03.490 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:00:03.561 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:00:03.561 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:00:04.220 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:00:04.221 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:00:04.221 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:00:04.221 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:00:04.222 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:00:04.222 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:00:04.222 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:04.222 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:04.222 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:00:04.223 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:04.223 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:00:04.223 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:04.223 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:04.224 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:00:04.224 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:04.224 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:00:04.225 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:04.225 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:00:04.225 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:04.225 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:04.226 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:04.226 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:04.226 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:00:04.226 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:04.227 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:04.227 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.914475_166.1", 2021-09-02T08:00:04.227 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:04.227 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:04.227 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:04.228 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.914475Z", 2021-09-02T08:00:04.229 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:04.229 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:04.229 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:04.229 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:04.230 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:00:04.230 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:04.230 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:04.230 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:04.231 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:04.231 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:04.231 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:04.231 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:04.232 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:04.232 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:04.232 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:04.232 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:04.232 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:00:04.233 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:04.233 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:04.233 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:04.233 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:04.234 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:04.234 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:04.234 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:04.235 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:04.243 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:04.243 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:04.244 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:00:04.244 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:04.244 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:00:04.244 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:00:04.245 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:00:04.245 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:04.245 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:00:04.246 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.914475_166.1 2021-09-02T08:00:09.247 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:09.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:09.249 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:00:09.298 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:00:09.299 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:00:09.399 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:00:09.399 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:00:09.400 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:00:09.400 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:00:09.400 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:00:09.400 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:00:09.401 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:09.401 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:09.401 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:00:09.401 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:09.402 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:00:09.402 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:09.402 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:09.402 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:00:09.402 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:09.403 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:00:09.403 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:09.403 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:00:09.403 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:09.404 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:09.404 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:09.404 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:09.404 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:00:09.405 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:09.405 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:09.405 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.911309_165.1", 2021-09-02T08:00:09.405 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:09.406 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:09.406 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:09.406 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.911309Z", 2021-09-02T08:00:09.406 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:09.407 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:09.407 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:09.407 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:09.407 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:00:09.408 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:09.408 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:09.408 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:09.408 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:09.409 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:09.409 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:09.409 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:09.409 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:09.410 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:09.410 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:09.410 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:09.410 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:00:09.410 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:09.411 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:09.411 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:09.411 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:09.411 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:09.412 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:09.412 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:09.412 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:09.412 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:09.412 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:09.413 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:00:09.413 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:09.413 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:00:09.413 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:00:09.414 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:00:09.414 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:09.414 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:00:09.415 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.911309_165.1 2021-09-02T08:00:14.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:14.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:14.420 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:00:14.470 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:00:14.470 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:00:14.546 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:00:14.547 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:00:14.547 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:00:14.547 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:00:14.548 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:00:14.548 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:00:14.548 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:14.549 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:14.549 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:00:14.549 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:14.549 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:00:14.549 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:14.550 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:14.550 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:00:14.550 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:14.550 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:00:14.551 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:14.551 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:00:14.551 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:14.551 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:14.552 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:14.552 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:14.552 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:00:14.552 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:14.553 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:14.553 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.914475_166.1", 2021-09-02T08:00:14.553 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:14.553 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:14.553 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:14.554 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.914475Z", 2021-09-02T08:00:14.554 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:14.554 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:14.554 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:14.555 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:14.555 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:00:14.555 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:14.555 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:14.556 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:14.556 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:14.556 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:14.556 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:14.557 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:14.557 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:14.557 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:14.557 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:14.557 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:14.558 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:00:14.558 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:14.558 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:14.559 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:14.559 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:14.559 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:14.559 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:14.560 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:14.560 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:14.560 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:14.560 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:14.561 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:00:14.561 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:14.562 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:00:14.562 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:00:14.562 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:00:14.563 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:14.563 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:00:14.565 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.914475_166.1 2021-09-02T08:00:19.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:19.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:19.569 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:00:19.614 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:00:19.615 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:00:19.696 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:00:19.696 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:00:19.697 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:00:19.697 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:00:19.697 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:00:19.698 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:00:19.698 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:19.698 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:19.698 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:00:19.699 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:19.700 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:00:19.700 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:19.700 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:19.701 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:00:19.701 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:19.701 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:00:19.701 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:19.702 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:00:19.702 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:19.703 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:19.703 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:19.703 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:19.703 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:00:19.704 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:19.704 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:19.704 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T08:00:19.704 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:19.705 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:19.705 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:19.705 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T08:00:19.705 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:19.706 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:19.706 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:19.706 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:19.706 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:00:19.707 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:19.707 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:19.707 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:19.707 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:19.708 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:19.708 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:19.708 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:19.708 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:19.709 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:19.709 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:19.709 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:19.709 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:00:19.710 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:19.710 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:19.710 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:19.710 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:19.711 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:19.711 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:19.711 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:19.711 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:19.711 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:19.712 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:19.712 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:00:19.712 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:19.712 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:00:19.713 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:00:19.713 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:00:19.713 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:19.713 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:00:19.717 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.103991_175.1 2021-09-02T08:00:24.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:24.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:24.720 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:00:24.813 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:00:24.813 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:00:25.091 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:00:25.091 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:00:25.091 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:00:25.092 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:00:25.092 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:00:25.092 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:00:25.093 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:25.093 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:25.093 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:00:25.093 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:25.094 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:00:25.094 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:25.094 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:25.094 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:00:25.095 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:25.095 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:00:25.095 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:25.095 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:00:25.095 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:25.096 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:25.096 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:25.096 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:25.096 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:00:25.097 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:25.097 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:25.097 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.249624_187.1", 2021-09-02T08:00:25.097 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:25.098 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:25.098 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:25.098 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.249624Z", 2021-09-02T08:00:25.098 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:25.098 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:25.099 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:25.099 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:25.099 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:00:25.099 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:25.100 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:25.100 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:25.100 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:25.101 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:25.101 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:25.101 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:25.101 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:25.102 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:25.102 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:25.102 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:25.102 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:00:25.102 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:25.103 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:25.103 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:25.103 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:25.103 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:25.104 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:25.104 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:25.104 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:25.104 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:25.104 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:25.105 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:00:25.105 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:25.105 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:00:25.106 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:00:25.106 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:00:25.106 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:25.106 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:00:25.108 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.249624_187.1 2021-09-02T08:00:30.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:30.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:30.111 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:00:30.158 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:00:30.159 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:00:30.397 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:00:30.398 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:00:30.398 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:00:30.399 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:00:30.399 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:00:30.399 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:00:30.399 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:30.400 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:30.400 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:00:30.400 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:30.400 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:00:30.401 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:30.401 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:30.401 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:00:30.401 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:30.401 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:00:30.402 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:30.402 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:00:30.402 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:30.402 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:30.405 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:30.405 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:30.406 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:00:30.406 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:30.406 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:30.406 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.914475_166.1", 2021-09-02T08:00:30.406 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:30.407 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:30.407 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:30.407 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.914475Z", 2021-09-02T08:00:30.407 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:30.408 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:30.408 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:30.408 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:30.408 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:00:30.409 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:30.409 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:30.409 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:30.409 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:30.410 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:30.410 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:30.410 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:30.410 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:30.411 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:30.411 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:30.411 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:30.411 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:00:30.411 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:30.412 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:30.412 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:30.412 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:30.412 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:30.413 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:30.413 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:30.413 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:30.413 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:30.413 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:30.414 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:00:30.414 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:30.414 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:00:30.414 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:00:30.415 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:00:30.415 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:30.415 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:00:30.416 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.914475_166.1 2021-09-02T08:00:35.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:35.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:35.419 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:00:35.509 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:00:35.509 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:00:35.809 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:00:35.809 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:00:35.809 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:00:35.810 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:00:35.810 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:00:35.810 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:00:35.810 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:35.811 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:35.811 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:00:35.811 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:35.811 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:00:35.811 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:35.812 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:35.812 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:00:35.812 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:35.812 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:00:35.813 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:35.813 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:00:35.813 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:35.814 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:35.814 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:35.814 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:35.814 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:00:35.815 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:35.815 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:35.815 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T08:00:35.815 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:35.815 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:35.816 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:35.816 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T08:00:35.816 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:35.816 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:35.817 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:35.817 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:35.817 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:00:35.817 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:35.817 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:35.818 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:35.818 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:35.818 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:35.818 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:35.819 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:35.819 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:35.819 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:35.819 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:35.820 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:35.822 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:00:35.823 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:35.823 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:35.823 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:35.824 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:35.824 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:35.824 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:35.824 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:35.825 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:35.825 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:35.825 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:35.825 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:00:35.826 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:35.826 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:00:35.826 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:00:35.826 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:00:35.827 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:35.827 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:00:35.828 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.906521_163.1 2021-09-02T08:00:40.830 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:40.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:40.832 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:00:40.879 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:00:40.880 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:00:41.272 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:00:41.273 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:00:41.273 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:00:41.273 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:00:41.274 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:00:41.274 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:00:41.274 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:41.274 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:41.275 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:00:41.275 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:41.275 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:00:41.275 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:41.276 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:41.276 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:00:41.277 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:41.277 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:00:41.277 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:41.278 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:00:41.278 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:41.279 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:41.279 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:41.279 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:41.279 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:00:41.280 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:41.280 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:41.281 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T08:00:41.281 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:41.281 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:41.281 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:41.281 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T08:00:41.282 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:41.282 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:41.282 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:41.282 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:41.283 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:00:41.283 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:41.283 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:41.283 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:41.284 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:41.284 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:41.284 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:41.284 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:41.284 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:41.285 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:41.285 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:41.285 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:41.285 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:00:41.286 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:41.286 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:41.286 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:41.287 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:41.287 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:41.287 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:41.288 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:41.288 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:41.289 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:41.289 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:41.290 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:00:41.290 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:41.291 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:00:41.291 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:00:41.291 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:00:41.292 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:41.292 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:00:41.293 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.906521_163.1 2021-09-02T08:00:46.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:46.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:46.297 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:00:46.350 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:00:46.350 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:00:46.427 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:00:46.428 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:00:46.428 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:00:46.428 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:00:46.428 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:00:46.429 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:00:46.429 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:46.429 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:46.429 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:00:46.430 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:46.430 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:00:46.430 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:46.430 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:46.430 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:00:46.431 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:46.431 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:00:46.431 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:46.431 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:00:46.432 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:46.432 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:46.432 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:46.432 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:46.432 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:00:46.433 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:46.433 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:46.433 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.911309_165.1", 2021-09-02T08:00:46.433 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:46.434 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:46.434 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:46.434 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.911309Z", 2021-09-02T08:00:46.434 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:46.435 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:46.435 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:46.435 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:46.435 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:00:46.435 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:46.436 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:46.436 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:46.436 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:46.436 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:46.436 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:46.437 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:46.437 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:46.437 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:46.437 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:46.438 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:46.438 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:00:46.438 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:46.438 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:46.438 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:46.439 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:46.439 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:46.439 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:46.439 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:46.440 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:46.440 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:46.440 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:46.440 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:00:46.440 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:46.441 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:00:46.441 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:00:46.441 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:00:46.441 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:46.442 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:00:46.443 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.911309_165.1 2021-09-02T08:00:51.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:51.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:51.446 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:00:51.491 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:00:51.492 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:00:51.562 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:00:51.562 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:00:51.563 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:00:51.563 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:00:51.563 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:00:51.563 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:00:51.564 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:51.564 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:51.564 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:00:51.564 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:51.564 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:00:51.565 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:51.565 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:51.565 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:00:51.565 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:51.566 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:00:51.566 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:51.566 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:00:51.566 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:51.567 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:51.567 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:51.567 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:51.567 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:00:51.567 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:51.568 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:51.568 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T08:00:51.568 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:51.568 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:51.569 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:51.569 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T08:00:51.569 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:51.569 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:51.570 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:51.570 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:51.570 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:00:51.570 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:51.572 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:51.573 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:51.573 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:51.573 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:51.574 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:51.574 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:51.574 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:51.574 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:51.575 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:51.576 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:51.576 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:00:51.577 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:51.577 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:51.577 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:51.577 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:51.578 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:51.578 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:51.578 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:51.578 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:51.579 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:51.579 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:51.579 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:00:51.579 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:51.580 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:00:51.580 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:00:51.580 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:00:51.580 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:51.580 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:00:51.582 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.103991_175.1 2021-09-02T08:00:56.583 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:56.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:00:56.585 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:00:56.634 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:00:56.634 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:00:56.998 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:00:56.998 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:00:56.998 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:00:56.999 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:00:56.999 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:00:56.999 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:00:56.999 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:57.000 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:57.000 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:00:57.000 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:57.000 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:00:57.001 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:57.001 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:57.001 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:00:57.001 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:57.002 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:00:57.002 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:57.002 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:00:57.002 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:57.003 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:57.003 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:57.003 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:57.003 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:00:57.005 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:57.005 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:57.005 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.990121_170.1", 2021-09-02T08:00:57.005 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:57.006 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:57.006 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:57.006 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.990121Z", 2021-09-02T08:00:57.006 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:57.007 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:57.007 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:57.007 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:57.007 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:00:57.007 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:57.008 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:57.008 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:57.008 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:57.008 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:57.009 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:57.009 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:57.009 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:57.009 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:57.010 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:57.010 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:00:57.010 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:00:57.010 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:00:57.010 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:00:57.011 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:00:57.011 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:00:57.011 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:00:57.011 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:00:57.012 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:00:57.012 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:00:57.012 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:57.012 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:57.013 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:00:57.013 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:00:57.013 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:00:57.013 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:00:57.013 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:00:57.014 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:00:57.014 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:00:57.015 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.990121_170.1 2021-09-02T08:01:02.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:02.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:02.017 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:01:02.105 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:01:02.106 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:01:02.283 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:01:02.283 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:01:02.283 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:01:02.284 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:01:02.284 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:01:02.284 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:01:02.284 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:02.285 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:02.285 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:01:02.285 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:02.285 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:01:02.286 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:02.286 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:02.286 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:01:02.286 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:02.287 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:01:02.287 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:02.287 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:01:02.287 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:02.288 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:02.288 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:02.288 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:02.288 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:01:02.288 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:02.289 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:02.289 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.985199_169.1", 2021-09-02T08:01:02.290 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:02.290 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:02.292 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:02.293 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.985199Z", 2021-09-02T08:01:02.293 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:02.293 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:02.293 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:02.294 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:02.294 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:01:02.294 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:02.294 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:02.295 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:02.295 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:02.295 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:02.295 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:02.296 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:02.296 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:02.296 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:02.296 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:02.296 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:02.297 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:01:02.297 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:02.297 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:02.297 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:02.298 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:02.298 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:02.298 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:02.298 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:02.299 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:02.299 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:02.299 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:02.299 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:01:02.300 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:02.300 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:01:02.300 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:01:02.300 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:01:02.301 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:02.301 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:01:02.303 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.985199_169.1 2021-09-02T08:01:07.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:07.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:07.306 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:01:07.407 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:01:07.408 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:01:07.987 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:01:07.987 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:01:07.987 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:01:07.988 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:01:07.988 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:01:07.988 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:01:07.988 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:07.989 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:07.989 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:01:07.989 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:07.989 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:01:07.990 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:07.990 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:07.990 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:01:07.990 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:07.991 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:01:07.991 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:07.991 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:01:07.991 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:07.991 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:07.992 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:07.992 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:07.992 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:01:07.992 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:07.993 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:07.993 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T08:01:07.993 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:07.993 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:07.993 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:07.994 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T08:01:07.994 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:07.994 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:07.994 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:07.995 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:07.995 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:01:07.995 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:07.995 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:07.996 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:07.996 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:07.996 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:07.996 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:07.997 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:07.997 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:07.997 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:07.997 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:07.997 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:07.998 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:01:07.998 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:07.998 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:07.998 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:07.999 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:07.999 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:07.999 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:07.999 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:07.999 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:08.000 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:08.000 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:08.000 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:01:08.000 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:08.001 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:01:08.001 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:01:08.001 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:01:08.001 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:08.001 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:01:08.003 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.103991_175.1 2021-09-02T08:01:13.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:13.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:13.006 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:01:13.055 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:01:13.056 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:01:13.133 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:01:13.134 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:01:13.134 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:01:13.134 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:01:13.134 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:01:13.135 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:01:13.135 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:13.135 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:13.135 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:01:13.135 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:13.136 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:01:13.136 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:13.136 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:13.136 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:01:13.137 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:13.137 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:01:13.137 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:13.137 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:01:13.137 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:13.138 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:13.138 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:13.138 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:13.138 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:01:13.139 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:13.139 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:13.139 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.129118_179.1", 2021-09-02T08:01:13.140 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:13.140 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:13.140 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:13.140 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.129118Z", 2021-09-02T08:01:13.141 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:13.141 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:13.141 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:13.141 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:13.141 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:01:13.142 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:13.142 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:13.143 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:13.143 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:13.143 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:13.143 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:13.144 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:13.144 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:13.144 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:13.144 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:13.144 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:13.145 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:01:13.145 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:13.145 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:13.145 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:13.146 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:13.146 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:13.146 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:13.146 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:13.147 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:13.147 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:13.147 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:13.147 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:01:13.147 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:13.148 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:01:13.148 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:01:13.148 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:01:13.148 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:13.148 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:01:13.150 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.129118_179.1 2021-09-02T08:01:18.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:18.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:18.152 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:01:18.197 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:01:18.198 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:01:18.268 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:01:18.268 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:01:18.268 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:01:18.269 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:01:18.269 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:01:18.269 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:01:18.269 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:18.270 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:18.270 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:01:18.270 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:18.270 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:01:18.271 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:18.271 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:18.271 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:01:18.271 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:18.272 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:01:18.273 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:18.273 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:01:18.273 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:18.273 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:18.274 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:18.274 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:18.274 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:01:18.274 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:18.275 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:18.275 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.929348_167.1", 2021-09-02T08:01:18.275 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:18.275 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:18.276 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:18.276 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.929348Z", 2021-09-02T08:01:18.276 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:18.276 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:18.277 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:18.277 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:18.277 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:01:18.277 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:18.278 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:18.278 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:18.278 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:18.278 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:18.279 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:18.279 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:18.279 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:18.279 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:18.279 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:18.280 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:18.280 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:01:18.280 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:18.280 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:18.281 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:18.281 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:18.281 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:18.281 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:18.282 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:18.282 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:18.282 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:18.282 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:18.283 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:01:18.283 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:18.283 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:01:18.283 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:01:18.283 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:01:18.284 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:18.284 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:01:18.285 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.929348_167.1 2021-09-02T08:01:23.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... FAIL 2021-09-02T08:01:23.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:23.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:23.290 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:01:23.340 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:01:23.340 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:01:23.389 INFO:teuthology.orchestra.run.smithi175.stderr:No --period given, using current period=3c6be1ae-90c2-49d2-9d54-568c49b66f2d 2021-09-02T08:01:23.410 INFO:teuthology.orchestra.run.smithi175.stdout:[ 2021-09-02T08:01:23.411 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:01:23.411 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:01:23.411 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:32:10.734248Z" 2021-09-02T08:01:23.411 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:01:23.411 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:01:23.412 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630569295.762188_292.1", 2021-09-02T08:01:23.412 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "2021-09-02T07:54:55.762188Z" 2021-09-02T08:01:23.412 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:01:23.412 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:01:23.413 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T08:01:23.413 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T08:01:23.413 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:01:23.413 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:01:23.414 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T08:01:23.414 INFO:teuthology.orchestra.run.smithi175.stdout: "last_update": "0.000000" 2021-09-02T08:01:23.414 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:01:23.414 INFO:teuthology.orchestra.run.smithi175.stdout:] 2021-09-02T08:01:23.415 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-02T08:01:23.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:23.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:23.416 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:01:23.463 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:01:23.463 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:01:23.549 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:01:23.550 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:01:23.550 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:01:23.550 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:01:23.551 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:01:23.551 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:01:23.552 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:23.552 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:23.553 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:01:23.553 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:23.554 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:01:23.554 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:23.554 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:23.555 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:01:23.555 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:23.556 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:01:23.556 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:23.557 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:01:23.557 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:23.557 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:23.558 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:23.558 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:23.559 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:01:23.559 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:23.560 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:23.560 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630569295.762188_292.1", 2021-09-02T08:01:23.561 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:23.561 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:23.561 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:23.562 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:54:55.762188Z", 2021-09-02T08:01:23.562 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:23.563 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:23.563 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:23.564 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:23.564 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:01:23.564 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:23.565 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:23.565 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:23.566 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:23.566 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:23.567 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:23.567 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:23.567 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:23.568 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:23.568 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:23.569 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:23.569 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:01:23.570 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:23.570 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:23.570 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:23.571 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:23.571 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:23.572 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:23.572 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:23.573 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:23.573 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:23.574 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:23.574 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:01:23.574 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:23.575 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:01:23.575 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:01:23.576 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:01:23.576 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:23.577 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:01:23.579 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-02T08:01:23.579 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-02T08:01:23.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:23.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:23.582 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:01:23.627 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:01:23.628 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:01:23.692 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T08:01:23.692 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": { 2021-09-02T08:01:23.692 INFO:teuthology.orchestra.run.smithi175.stdout: "info": { 2021-09-02T08:01:23.693 INFO:teuthology.orchestra.run.smithi175.stdout: "status": "sync", 2021-09-02T08:01:23.693 INFO:teuthology.orchestra.run.smithi175.stdout: "num_shards": 4, 2021-09-02T08:01:23.693 INFO:teuthology.orchestra.run.smithi175.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:01:23.693 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T08:01:23.694 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:01:23.694 INFO:teuthology.orchestra.run.smithi175.stdout: "markers": [ 2021-09-02T08:01:23.694 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:01:23.694 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 0, 2021-09-02T08:01:23.695 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T08:01:23.695 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T08:01:23.695 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:01:23.695 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T08:01:23.695 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 1, 2021-09-02T08:01:23.696 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T08:01:23.696 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:01:23.696 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T08:01:23.697 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:01:23.697 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:01:23.698 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:01:23.699 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 1, 2021-09-02T08:01:23.699 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T08:01:23.699 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T08:01:23.699 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "1_1630569295.762188_292.1", 2021-09-02T08:01:23.700 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T08:01:23.700 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T08:01:23.700 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T08:01:23.700 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "2021-09-02T07:54:55.762188Z", 2021-09-02T08:01:23.700 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T08:01:23.701 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:01:23.701 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:01:23.701 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:01:23.701 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 2, 2021-09-02T08:01:23.702 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T08:01:23.702 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T08:01:23.702 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T08:01:23.702 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T08:01:23.702 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T08:01:23.703 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T08:01:23.703 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T08:01:23.703 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T08:01:23.703 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:01:23.704 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:01:23.704 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:01:23.705 INFO:teuthology.orchestra.run.smithi175.stdout: "key": 3, 2021-09-02T08:01:23.705 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T08:01:23.705 INFO:teuthology.orchestra.run.smithi175.stdout: "state": 1, 2021-09-02T08:01:23.706 INFO:teuthology.orchestra.run.smithi175.stdout: "marker": "", 2021-09-02T08:01:23.707 INFO:teuthology.orchestra.run.smithi175.stdout: "next_step_marker": "", 2021-09-02T08:01:23.707 INFO:teuthology.orchestra.run.smithi175.stdout: "total_entries": 0, 2021-09-02T08:01:23.707 INFO:teuthology.orchestra.run.smithi175.stdout: "pos": 0, 2021-09-02T08:01:23.707 INFO:teuthology.orchestra.run.smithi175.stdout: "timestamp": "0.000000", 2021-09-02T08:01:23.708 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T08:01:23.708 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:01:23.708 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:01:23.708 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T08:01:23.709 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:01:23.709 INFO:teuthology.orchestra.run.smithi175.stdout: "full_sync": { 2021-09-02T08:01:23.709 INFO:teuthology.orchestra.run.smithi175.stdout: "total": 1, 2021-09-02T08:01:23.709 INFO:teuthology.orchestra.run.smithi175.stdout: "complete": 1 2021-09-02T08:01:23.710 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:01:23.710 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T08:01:23.711 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-02T08:01:23.712 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-02T08:01:23.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:23.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:23.713 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:01:23.759 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:01:23.760 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:01:23.843 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:01:23.844 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:01:23.844 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:01:23.844 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:01:23.844 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:01:23.845 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:01:23.845 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:23.845 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:23.845 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:01:23.846 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:23.846 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:01:23.846 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:23.846 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:23.847 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:01:23.847 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:23.847 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:01:23.847 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:23.848 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:01:23.848 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:23.848 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:23.848 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:23.848 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:23.849 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:01:23.849 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:23.849 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:23.849 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T08:01:23.850 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:23.850 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:23.850 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:23.850 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T08:01:23.851 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:23.851 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:23.851 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:23.851 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:23.851 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:01:23.852 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:23.852 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:23.852 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:23.852 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:23.853 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:23.853 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:23.853 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:23.853 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:23.854 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:23.854 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:23.854 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:23.854 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:01:23.855 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:23.855 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:23.855 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:23.855 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:23.855 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:23.856 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:23.856 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:23.856 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:23.856 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:23.857 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:23.857 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:01:23.857 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:23.857 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:01:23.857 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:01:23.858 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:01:23.858 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:23.858 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:01:23.859 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.906521_163.1 2021-09-02T08:01:28.860 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:28.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:28.862 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:01:28.925 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:01:28.925 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:01:28.995 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:01:28.995 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:01:28.996 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:01:28.996 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:01:28.996 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:01:28.996 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:01:28.997 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:28.997 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:28.997 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:01:28.997 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:28.998 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:01:28.998 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:28.998 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:28.999 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:01:28.999 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:28.999 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:01:28.999 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:29.000 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:01:29.001 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:29.001 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:29.002 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:29.002 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:29.002 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:01:29.002 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:29.003 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:29.003 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.043408_172.1", 2021-09-02T08:01:29.003 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:29.003 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:29.004 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:29.004 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.043408Z", 2021-09-02T08:01:29.004 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:29.004 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:29.004 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:29.005 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:29.005 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:01:29.005 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:29.005 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:29.006 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:29.006 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:29.006 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:29.007 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:29.008 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:29.008 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:29.008 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:29.008 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:29.008 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:29.009 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:01:29.009 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:29.009 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:29.009 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:29.010 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:29.010 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:29.010 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:29.010 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:29.011 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:29.011 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:29.012 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:29.012 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:01:29.012 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:29.012 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:01:29.012 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:01:29.013 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:01:29.013 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:29.013 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:01:29.014 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.043408_172.1 2021-09-02T08:01:34.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:34.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:34.017 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:01:34.095 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:01:34.095 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:01:34.489 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:01:34.490 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:01:34.490 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:01:34.490 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:01:34.490 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:01:34.491 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:01:34.491 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:34.491 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:34.492 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:01:34.492 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:34.492 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:01:34.492 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:34.492 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:34.493 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:01:34.493 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:34.493 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:01:34.493 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:34.494 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:01:34.494 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:34.494 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:34.494 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:34.495 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:34.495 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:01:34.495 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:34.496 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:34.496 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T08:01:34.496 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:34.496 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:34.496 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:34.497 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T08:01:34.497 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:34.497 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:34.497 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:34.498 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:34.498 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:01:34.498 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:34.498 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:34.499 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:34.499 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:34.499 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:34.499 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:34.500 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:34.500 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:34.500 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:34.500 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:34.501 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:34.501 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:01:34.501 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:34.501 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:34.501 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:34.502 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:34.502 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:34.502 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:34.503 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:34.503 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:34.503 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:34.503 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:34.504 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:01:34.504 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:34.504 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:01:34.504 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:01:34.505 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:01:34.505 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:34.505 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:01:34.506 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.103991_175.1 2021-09-02T08:01:39.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:39.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:39.510 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:01:39.565 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:01:39.565 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:01:39.702 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:01:39.703 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:01:39.703 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:01:39.703 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:01:39.704 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:01:39.704 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:01:39.704 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:39.704 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:39.705 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:01:39.705 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:39.705 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:01:39.705 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:39.706 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:39.706 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:01:39.706 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:39.706 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:01:39.707 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:39.707 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:01:39.707 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:39.707 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:39.707 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:39.708 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:39.708 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:01:39.708 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:39.708 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:39.709 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T08:01:39.709 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:39.709 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:39.709 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:39.710 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T08:01:39.710 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:39.710 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:39.710 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:39.711 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:39.711 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:01:39.711 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:39.711 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:39.711 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:39.712 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:39.712 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:39.712 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:39.712 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:39.713 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:39.713 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:39.713 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:39.713 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:39.714 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:01:39.714 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:39.714 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:39.714 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:39.715 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:39.715 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:39.715 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:39.715 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:39.716 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:39.716 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:39.716 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:39.716 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:01:39.716 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:39.717 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:01:39.717 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:01:39.717 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:01:39.717 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:39.718 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:01:39.719 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.103991_175.1 2021-09-02T08:01:44.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:44.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:44.722 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:01:44.801 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:01:44.802 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:01:45.319 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:01:45.319 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:01:45.320 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:01:45.320 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:01:45.320 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:01:45.320 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:01:45.321 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:45.321 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:45.321 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:01:45.321 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:45.322 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:01:45.322 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:45.322 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:45.322 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:01:45.322 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:45.323 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:01:45.323 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:45.323 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:01:45.323 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:45.324 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:45.324 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:45.324 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:45.325 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:01:45.325 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:45.325 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:45.325 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.043408_172.1", 2021-09-02T08:01:45.326 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:45.326 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:45.326 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:45.326 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.043408Z", 2021-09-02T08:01:45.326 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:45.327 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:45.327 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:45.327 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:45.327 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:01:45.328 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:45.328 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:45.328 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:45.328 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:45.329 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:45.329 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:45.329 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:45.329 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:45.329 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:45.330 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:45.330 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:45.330 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:01:45.330 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:45.331 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:45.331 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:45.331 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:45.331 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:45.331 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:45.332 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:45.332 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:45.332 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:45.333 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:45.333 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:01:45.333 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:45.333 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:01:45.334 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:01:45.334 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:01:45.334 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:45.334 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:01:45.335 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.043408_172.1 2021-09-02T08:01:50.336 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:50.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:50.337 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:01:50.429 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:01:50.430 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:01:50.908 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:01:50.908 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:01:50.909 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:01:50.909 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:01:50.909 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:01:50.909 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:01:50.910 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:50.910 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:50.910 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:01:50.910 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:50.910 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:01:50.911 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:50.911 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:50.911 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:01:50.911 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:50.912 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:01:50.912 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:50.912 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:01:50.912 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:50.912 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:50.913 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:50.913 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:50.913 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:01:50.914 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:50.914 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:50.914 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T08:01:50.914 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:50.915 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:50.915 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:50.915 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T08:01:50.915 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:50.915 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:50.916 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:50.917 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:50.917 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:01:50.917 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:50.917 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:50.918 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:50.918 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:50.918 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:50.919 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:50.919 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:50.919 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:50.919 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:50.919 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:50.920 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:50.920 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:01:50.920 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:50.920 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:50.921 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:50.921 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:50.921 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:50.921 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:50.921 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:50.922 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:50.922 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:50.922 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:50.922 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:01:50.923 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:50.923 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:01:50.923 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:01:50.923 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:01:50.923 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:50.924 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:01:50.925 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.103991_175.1 2021-09-02T08:01:55.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:55.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:01:55.928 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:01:55.974 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:01:55.975 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:01:56.093 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:01:56.093 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:01:56.093 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:01:56.094 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:01:56.094 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:01:56.094 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:01:56.094 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:56.095 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:56.095 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:01:56.095 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:56.095 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:01:56.096 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:56.096 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:56.096 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:01:56.096 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:56.096 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:01:56.097 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:56.097 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:01:56.097 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:56.097 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:56.098 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:56.098 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:56.098 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:01:56.098 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:56.098 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:56.099 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.985199_169.1", 2021-09-02T08:01:56.099 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:56.099 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:56.099 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:56.100 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.985199Z", 2021-09-02T08:01:56.100 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:56.100 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:56.100 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:56.100 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:56.101 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:01:56.101 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:56.101 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:56.101 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:56.101 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:56.102 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:56.102 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:56.102 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:56.102 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:56.103 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:56.103 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:56.103 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:01:56.103 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:01:56.103 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:01:56.104 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:01:56.104 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:01:56.104 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:01:56.104 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:01:56.105 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:01:56.105 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:01:56.105 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:01:56.105 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:56.106 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:56.106 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:01:56.106 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:01:56.106 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:01:56.106 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:01:56.107 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:01:56.107 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:01:56.107 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:01:56.108 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.985199_169.1 2021-09-02T08:02:01.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:01.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:01.110 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:02:01.154 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:02:01.155 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:02:01.231 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:02:01.231 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:02:01.232 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:02:01.232 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:02:01.232 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:02:01.233 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:02:01.233 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:01.233 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:01.233 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:02:01.234 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:01.234 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:02:01.234 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:01.235 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:01.235 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:02:01.235 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:01.235 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:02:01.235 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:01.236 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:02:01.236 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:01.236 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:01.236 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:01.237 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:01.237 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:02:01.237 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:01.237 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:01.238 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T08:02:01.238 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:01.238 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:01.238 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:01.239 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T08:02:01.239 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:01.239 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:01.239 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:01.240 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:01.240 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:02:01.240 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:01.240 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:01.240 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:01.241 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:01.241 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:01.242 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:01.242 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:01.242 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:01.242 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:01.243 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:01.243 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:01.243 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:02:01.243 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:01.244 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:01.244 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:01.245 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:01.245 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:01.245 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:01.245 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:01.246 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:01.246 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:01.246 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:01.246 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:02:01.247 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:01.247 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:02:01.247 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:02:01.247 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:02:01.247 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:01.248 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:02:01.249 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.910082_164.1 2021-09-02T08:02:06.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:06.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:06.252 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:02:06.295 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:02:06.295 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:02:06.370 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:02:06.370 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:02:06.371 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:02:06.371 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:02:06.371 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:02:06.371 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:02:06.372 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:06.372 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:06.372 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:02:06.372 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:06.372 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:02:06.373 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:06.373 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:06.373 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:02:06.373 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:06.374 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:02:06.374 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:06.374 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:02:06.374 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:06.374 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:06.375 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:06.375 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:06.375 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:02:06.375 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:06.376 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:06.376 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.914475_166.1", 2021-09-02T08:02:06.376 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:06.376 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:06.376 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:06.377 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.914475Z", 2021-09-02T08:02:06.377 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:06.377 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:06.377 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:06.377 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:06.378 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:02:06.378 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:06.378 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:06.378 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:06.379 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:06.379 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:06.379 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:06.379 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:06.380 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:06.380 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:06.380 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:06.380 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:06.380 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:02:06.381 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:06.381 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:06.381 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:06.381 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:06.382 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:06.382 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:06.382 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:06.382 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:06.382 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:06.383 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:06.383 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:02:06.383 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:06.383 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:02:06.384 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:02:06.384 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:02:06.384 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:06.384 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:02:06.385 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.914475_166.1 2021-09-02T08:02:11.387 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:11.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:11.389 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:02:11.472 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:02:11.473 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:02:11.893 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:02:11.894 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:02:11.894 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:02:11.894 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:02:11.895 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:02:11.895 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:02:11.895 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:11.895 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:11.895 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:02:11.896 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:11.896 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:02:11.896 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:11.896 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:11.897 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:02:11.897 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:11.897 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:02:11.897 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:11.897 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:02:11.898 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:11.898 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:11.898 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:11.898 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:11.899 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:02:11.899 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:11.899 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:11.899 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.914475_166.1", 2021-09-02T08:02:11.899 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:11.900 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:11.900 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:11.900 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.914475Z", 2021-09-02T08:02:11.900 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:11.901 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:11.901 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:11.901 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:11.901 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:02:11.901 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:11.902 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:11.902 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:11.902 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:11.902 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:11.903 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:11.903 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:11.903 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:11.903 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:11.903 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:11.904 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:11.904 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:02:11.904 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:11.904 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:11.905 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:11.905 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:11.905 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:11.905 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:11.905 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:11.906 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:11.906 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:11.906 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:11.906 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:02:11.907 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:11.907 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:02:11.907 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:02:11.907 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:02:11.907 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:11.908 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:02:11.909 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.914475_166.1 2021-09-02T08:02:16.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:16.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:16.912 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:02:16.962 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:02:16.962 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:02:17.124 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:02:17.125 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:02:17.125 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:02:17.125 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:02:17.125 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:02:17.126 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:02:17.126 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:17.126 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:17.126 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:02:17.126 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:17.127 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:02:17.127 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:17.127 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:17.127 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:02:17.128 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:17.128 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:02:17.128 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:17.128 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:02:17.128 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:17.129 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:17.129 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:17.129 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:17.129 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:02:17.129 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:17.130 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:17.130 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.162463_181.1", 2021-09-02T08:02:17.130 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:17.130 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:17.131 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:17.131 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.162463Z", 2021-09-02T08:02:17.131 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:17.131 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:17.132 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:17.132 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:17.132 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:02:17.132 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:17.132 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:17.133 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:17.133 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:17.133 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:17.133 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:17.133 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:17.134 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:17.134 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:17.134 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:17.134 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:17.135 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:02:17.135 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:17.135 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:17.135 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:17.136 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:17.136 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:17.136 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:17.136 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:17.136 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:17.137 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:17.137 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:17.137 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:02:17.137 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:17.138 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:02:17.138 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:02:17.138 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:02:17.138 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:17.138 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:02:17.139 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.162463_181.1 2021-09-02T08:02:22.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:22.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:22.141 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:02:22.242 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:02:22.243 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:02:22.622 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:02:22.623 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:02:22.623 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:02:22.623 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:02:22.624 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:02:22.624 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:02:22.624 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:22.624 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:22.625 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:02:22.625 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:22.625 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:02:22.625 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:22.626 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:22.626 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:02:22.626 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:22.626 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:02:22.627 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:22.627 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:02:22.627 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:22.627 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:22.627 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:22.628 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:22.628 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:02:22.628 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:22.628 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:22.629 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T08:02:22.629 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:22.629 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:22.629 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:22.629 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T08:02:22.630 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:22.630 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:22.630 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:22.630 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:22.631 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:02:22.631 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:22.631 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:22.631 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:22.632 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:22.632 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:22.633 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:22.633 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:22.633 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:22.633 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:22.634 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:22.634 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:22.634 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:02:22.634 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:22.635 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:22.635 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:22.635 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:22.635 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:22.636 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:22.636 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:22.636 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:22.636 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:22.637 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:22.637 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:02:22.637 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:22.637 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:02:22.637 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:02:22.638 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:02:22.638 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:22.638 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:02:22.639 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.875533_162.1 2021-09-02T08:02:27.641 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:27.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:27.643 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:02:27.697 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:02:27.697 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:02:28.256 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:02:28.256 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:02:28.256 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:02:28.257 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:02:28.257 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:02:28.257 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:02:28.257 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:28.258 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:28.258 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:02:28.258 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:28.259 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:02:28.259 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:28.259 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:28.259 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:02:28.260 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:28.260 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:02:28.260 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:28.260 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:02:28.261 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:28.261 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:28.261 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:28.261 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:28.261 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:02:28.262 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:28.262 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:28.262 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.039849_171.1", 2021-09-02T08:02:28.262 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:28.263 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:28.263 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:28.263 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.039849Z", 2021-09-02T08:02:28.263 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:28.263 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:28.264 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:28.264 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:28.264 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:02:28.264 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:28.265 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:28.265 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:28.265 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:28.265 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:28.265 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:28.266 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:28.266 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:28.266 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:28.266 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:28.267 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:28.267 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:02:28.267 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:28.267 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:28.267 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:28.268 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:28.268 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:28.268 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:28.268 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:28.269 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:28.269 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:28.269 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:28.269 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:02:28.269 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:28.270 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:02:28.270 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:02:28.270 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:02:28.270 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:28.271 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:02:28.271 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.039849_171.1 2021-09-02T08:02:33.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:33.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:33.275 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:02:33.325 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:02:33.326 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:02:33.400 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:02:33.400 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:02:33.400 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:02:33.401 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:02:33.401 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:02:33.401 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:02:33.401 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:33.401 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:33.402 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:02:33.402 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:33.402 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:02:33.402 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:33.403 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:33.403 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:02:33.403 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:33.403 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:02:33.404 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:33.404 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:02:33.404 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:33.404 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:33.404 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:33.405 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:33.405 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:02:33.405 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:33.405 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:33.406 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T08:02:33.406 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:33.406 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:33.406 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:33.407 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T08:02:33.407 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:33.407 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:33.407 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:33.408 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:33.408 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:02:33.408 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:33.408 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:33.408 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:33.409 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:33.409 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:33.409 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:33.409 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:33.409 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:33.410 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:33.410 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:33.410 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:33.411 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:02:33.411 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:33.411 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:33.411 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:33.412 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:33.412 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:33.412 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:33.412 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:33.413 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:33.413 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:33.413 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:33.413 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:02:33.413 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:33.414 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:02:33.414 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:02:33.414 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:02:33.414 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:33.415 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:02:33.416 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.906521_163.1 2021-09-02T08:02:38.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:38.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:38.419 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:02:38.464 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:02:38.465 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:02:38.548 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:02:38.548 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:02:38.549 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:02:38.549 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:02:38.549 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:02:38.549 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:02:38.550 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:38.550 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:38.550 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:02:38.550 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:38.551 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:02:38.551 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:38.551 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:38.552 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:02:38.552 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:38.552 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:02:38.552 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:38.553 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:02:38.553 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:38.553 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:38.553 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:38.553 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:38.554 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:02:38.554 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:38.554 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:38.554 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.932719_168.1", 2021-09-02T08:02:38.555 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:38.555 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:38.555 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:38.555 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.932719Z", 2021-09-02T08:02:38.556 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:38.556 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:38.556 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:38.556 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:38.556 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:02:38.557 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:38.557 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:38.557 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:38.557 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:38.558 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:38.558 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:38.558 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:38.559 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:38.559 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:38.559 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:38.559 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:38.560 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:02:38.560 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:38.560 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:38.560 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:38.561 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:38.561 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:38.561 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:38.561 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:38.562 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:38.562 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:38.562 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:38.562 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:02:38.563 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:38.563 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:02:38.563 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:02:38.563 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:02:38.563 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:38.564 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:02:38.564 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.932719_168.1 2021-09-02T08:02:43.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:43.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:43.567 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:02:43.639 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:02:43.639 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:02:43.877 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:02:43.878 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:02:43.878 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:02:43.878 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:02:43.878 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:02:43.879 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:02:43.879 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:43.879 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:43.879 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:02:43.879 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:43.880 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:02:43.880 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:43.880 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:43.880 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:02:43.881 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:43.881 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:02:43.881 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:43.881 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:02:43.882 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:43.882 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:43.882 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:43.882 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:43.882 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:02:43.883 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:43.883 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:43.883 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.129118_179.1", 2021-09-02T08:02:43.883 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:43.884 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:43.884 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:43.884 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.129118Z", 2021-09-02T08:02:43.884 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:43.884 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:43.885 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:43.885 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:43.885 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:02:43.886 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:43.886 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:43.886 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:43.886 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:43.887 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:43.887 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:43.887 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:43.887 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:43.888 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:43.888 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:43.888 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:43.888 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:02:43.888 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:43.889 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:43.889 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:43.889 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:43.889 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:43.889 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:43.890 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:43.890 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:43.890 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:43.890 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:43.891 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:02:43.891 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:43.891 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:02:43.891 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:02:43.891 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:02:43.892 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:43.892 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:02:43.893 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.129118_179.1 2021-09-02T08:02:48.894 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:48.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:48.896 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:02:48.949 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:02:48.949 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:02:49.151 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:02:49.151 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:02:49.151 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:02:49.152 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:02:49.152 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:02:49.152 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:02:49.153 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:49.153 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:49.153 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:02:49.153 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:49.154 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:02:49.154 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:49.154 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:49.155 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:02:49.155 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:49.155 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:02:49.156 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:49.156 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:02:49.156 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:49.156 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:49.157 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:49.157 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:49.157 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:02:49.157 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:49.158 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:49.158 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T08:02:49.158 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:49.160 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:49.160 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:49.160 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T08:02:49.161 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:49.161 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:49.162 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:49.162 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:49.163 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:02:49.163 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:49.163 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:49.164 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:49.164 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:49.165 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:49.165 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:49.166 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:49.166 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:49.167 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:49.167 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:49.167 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:49.168 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:02:49.168 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:49.169 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:49.169 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:49.169 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:49.170 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:49.170 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:49.170 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:49.171 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:49.171 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:49.172 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:49.172 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:02:49.172 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:49.173 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:02:49.173 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:02:49.173 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:02:49.173 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:49.173 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:02:49.175 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.906521_163.1 2021-09-02T08:02:54.176 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:54.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:54.178 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:02:54.266 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:02:54.267 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:02:54.492 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:02:54.492 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:02:54.493 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:02:54.493 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:02:54.493 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:02:54.493 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:02:54.494 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:54.494 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:54.494 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:02:54.494 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:54.495 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:02:54.495 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:54.495 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:54.495 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:02:54.496 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:54.496 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:02:54.496 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:54.496 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:02:54.496 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:54.497 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:54.497 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:54.497 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:54.497 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:02:54.498 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:54.498 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:54.498 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T08:02:54.498 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:54.499 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:54.499 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:54.499 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T08:02:54.499 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:54.500 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:54.500 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:54.500 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:54.500 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:02:54.500 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:54.501 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:54.501 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:54.501 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:54.501 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:54.502 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:54.502 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:54.502 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:54.502 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:54.503 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:54.503 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:02:54.503 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:02:54.503 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:02:54.503 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:02:54.504 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:02:54.504 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:02:54.504 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:02:54.504 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:02:54.505 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:02:54.505 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:02:54.505 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:54.505 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:54.506 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:02:54.506 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:02:54.506 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:02:54.506 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:02:54.507 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:02:54.507 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:02:54.507 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:02:54.508 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.103991_175.1 2021-09-02T08:02:59.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:59.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:02:59.511 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:02:59.597 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:02:59.597 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:03:00.108 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:03:00.108 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:03:00.108 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:03:00.109 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:03:00.109 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:03:00.109 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:03:00.109 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:00.110 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:00.110 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:03:00.110 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:00.110 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:03:00.111 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:00.111 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:00.111 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:03:00.112 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:00.112 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:03:00.112 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:00.112 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:03:00.113 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:00.113 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:00.113 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:00.113 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:00.114 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:03:00.114 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:00.114 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:00.114 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T08:03:00.114 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:00.115 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:00.115 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:00.115 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T08:03:00.115 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:00.116 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:00.116 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:00.116 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:00.116 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:03:00.116 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:00.117 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:00.117 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:00.117 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:00.117 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:00.117 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:00.118 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:00.118 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:00.118 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:00.118 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:00.119 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:00.119 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:03:00.119 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:00.119 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:00.119 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:00.120 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:00.121 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:00.122 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:00.122 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:00.122 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:00.123 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:00.123 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:00.124 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:03:00.124 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:00.124 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:03:00.125 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:03:00.125 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:03:00.125 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:00.126 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:03:00.128 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.103991_175.1 2021-09-02T08:03:05.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:05.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:05.132 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:03:05.181 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:03:05.181 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:03:05.273 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:03:05.273 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:03:05.274 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:03:05.274 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:03:05.274 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:03:05.274 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:03:05.275 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:05.275 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:05.275 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:03:05.275 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:05.276 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:03:05.276 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:05.276 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:05.276 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:03:05.277 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:05.277 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:03:05.278 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:05.278 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:03:05.278 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:05.279 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:05.279 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:05.279 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:05.279 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:03:05.279 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:05.280 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:05.280 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.107268_176.1", 2021-09-02T08:03:05.280 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:05.280 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:05.281 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:05.281 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.107268Z", 2021-09-02T08:03:05.281 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:05.281 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:05.282 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:05.282 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:05.282 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:03:05.283 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:05.283 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:05.283 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:05.283 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:05.283 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:05.284 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:05.284 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:05.284 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:05.285 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:05.285 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:05.285 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:05.285 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:03:05.286 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:05.286 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:05.286 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:05.286 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:05.286 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:05.287 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:05.287 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:05.287 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:05.287 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:05.288 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:05.288 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:03:05.288 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:05.288 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:03:05.289 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:03:05.289 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:03:05.289 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:05.289 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:03:05.291 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.107268_176.1 2021-09-02T08:03:10.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:10.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:10.294 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:03:10.338 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:03:10.339 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:03:10.412 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:03:10.413 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:03:10.413 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:03:10.414 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:03:10.414 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:03:10.414 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:03:10.414 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:10.415 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:10.415 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:03:10.415 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:10.415 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:03:10.416 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:10.416 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:10.416 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:03:10.416 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:10.416 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:03:10.417 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:10.417 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:03:10.417 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:10.417 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:10.418 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:10.418 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:10.418 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:03:10.419 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:10.419 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:10.419 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T08:03:10.420 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:10.420 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:10.421 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:10.421 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T08:03:10.421 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:10.422 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:10.422 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:10.423 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:10.423 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:03:10.424 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:10.424 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:10.424 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:10.425 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:10.425 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:10.426 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:10.426 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:10.426 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:10.427 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:10.427 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:10.428 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:10.428 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:03:10.429 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:10.429 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:10.429 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:10.429 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:10.430 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:10.430 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:10.430 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:10.430 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:10.431 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:10.431 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:10.431 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:03:10.431 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:10.432 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:03:10.432 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:03:10.432 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:03:10.432 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:10.433 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:03:10.434 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.910082_164.1 2021-09-02T08:03:15.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:15.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:15.437 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:03:15.520 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:03:15.521 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:03:15.670 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:03:15.670 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:03:15.670 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:03:15.671 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:03:15.671 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:03:15.671 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:03:15.671 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:15.671 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:15.672 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:03:15.672 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:15.672 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:03:15.672 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:15.673 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:15.673 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:03:15.673 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:15.673 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:03:15.673 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:15.674 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:03:15.674 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:15.674 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:15.674 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:15.675 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:15.675 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:03:15.675 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:15.675 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:15.676 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T08:03:15.676 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:15.676 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:15.676 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:15.676 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T08:03:15.677 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:15.677 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:15.677 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:15.677 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:15.678 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:03:15.678 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:15.678 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:15.678 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:15.678 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:15.679 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:15.679 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:15.679 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:15.679 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:15.680 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:15.680 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:15.680 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:15.680 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:03:15.681 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:15.681 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:15.681 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:15.681 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:15.681 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:15.682 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:15.682 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:15.682 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:15.682 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:15.683 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:15.683 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:03:15.683 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:15.683 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:03:15.683 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:03:15.684 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:03:15.684 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:15.684 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:03:15.685 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.875533_162.1 2021-09-02T08:03:20.686 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:20.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:20.688 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:03:20.731 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:03:20.731 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:03:20.879 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:03:20.879 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:03:20.879 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:03:20.880 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:03:20.880 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:03:20.880 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:03:20.880 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:20.881 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:20.881 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:03:20.881 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:20.881 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:03:20.882 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:20.882 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:20.882 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:03:20.882 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:20.883 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:03:20.883 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:20.883 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:03:20.883 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:20.884 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:20.884 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:20.884 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:20.884 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:03:20.884 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:20.885 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:20.885 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T08:03:20.885 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:20.885 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:20.886 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:20.886 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T08:03:20.886 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:20.886 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:20.887 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:20.887 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:20.887 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:03:20.887 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:20.887 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:20.888 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:20.888 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:20.888 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:20.888 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:20.889 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:20.889 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:20.889 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:20.889 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:20.889 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:20.890 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:03:20.890 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:20.890 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:20.890 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:20.891 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:20.891 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:20.891 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:20.891 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:20.891 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:20.892 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:20.892 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:20.892 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:03:20.892 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:20.893 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:03:20.893 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:03:20.893 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:03:20.893 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:20.893 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:03:20.895 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.910082_164.1 2021-09-02T08:03:25.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:25.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:25.899 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:03:25.983 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:03:25.983 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:03:26.383 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:03:26.383 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:03:26.383 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:03:26.384 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:03:26.384 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:03:26.384 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:03:26.384 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:26.385 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:26.385 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:03:26.385 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:26.386 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:03:26.386 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:26.386 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:26.386 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:03:26.387 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:26.387 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:03:26.387 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:26.387 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:03:26.388 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:26.388 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:26.388 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:26.388 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:26.388 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:03:26.389 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:26.389 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:26.389 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T08:03:26.389 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:26.390 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:26.390 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:26.390 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T08:03:26.390 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:26.390 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:26.391 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:26.391 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:26.391 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:03:26.391 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:26.392 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:26.392 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:26.392 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:26.392 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:26.392 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:26.393 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:26.393 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:26.393 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:26.393 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:26.394 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:26.394 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:03:26.394 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:26.394 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:26.395 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:26.395 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:26.395 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:26.396 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:26.396 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:26.396 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:26.396 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:26.396 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:26.397 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:03:26.397 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:26.397 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:03:26.397 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:03:26.398 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:03:26.398 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:26.398 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:03:26.399 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.875533_162.1 2021-09-02T08:03:31.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:31.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:31.402 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:03:31.491 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:03:31.492 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:03:31.996 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:03:31.997 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:03:31.997 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:03:31.997 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:03:31.997 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:03:31.998 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:03:31.998 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:31.998 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:31.999 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:03:31.999 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:31.999 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:03:31.999 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:31.999 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:32.000 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:03:32.000 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:32.000 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:03:32.000 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:32.001 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:03:32.001 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:32.001 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:32.001 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:32.002 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:32.002 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:03:32.002 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:32.002 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:32.002 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.929348_167.1", 2021-09-02T08:03:32.003 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:32.003 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:32.003 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:32.003 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.929348Z", 2021-09-02T08:03:32.004 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:32.004 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:32.004 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:32.004 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:32.004 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:03:32.005 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:32.005 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:32.005 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:32.005 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:32.006 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:32.006 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:32.006 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:32.006 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:32.007 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:32.007 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:32.007 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:32.008 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:03:32.008 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:32.008 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:32.008 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:32.008 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:32.009 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:32.009 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:32.009 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:32.009 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:32.010 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:32.010 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:32.010 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:03:32.010 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:32.010 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:03:32.011 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:03:32.011 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:03:32.011 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:32.011 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:03:32.013 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.929348_167.1 2021-09-02T08:03:37.014 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:37.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:37.016 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:03:37.100 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:03:37.101 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:03:37.596 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:03:37.596 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:03:37.597 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:03:37.597 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:03:37.597 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:03:37.597 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:03:37.597 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:37.598 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:37.598 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:03:37.598 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:37.598 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:03:37.599 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:37.599 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:37.599 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:03:37.599 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:37.600 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:03:37.600 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:37.600 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:03:37.600 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:37.600 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:37.601 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:37.601 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:37.601 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:03:37.601 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:37.601 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:37.602 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T08:03:37.602 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:37.602 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:37.602 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:37.603 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T08:03:37.603 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:37.603 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:37.603 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:37.603 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:37.604 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:03:37.604 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:37.604 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:37.604 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:37.605 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:37.605 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:37.606 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:37.607 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:37.607 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:37.607 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:37.607 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:37.608 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:37.608 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:03:37.608 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:37.608 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:37.609 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:37.609 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:37.609 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:37.609 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:37.610 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:37.610 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:37.610 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:37.610 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:37.611 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:03:37.611 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:37.611 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:03:37.611 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:03:37.612 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:03:37.612 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:37.612 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:03:37.613 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.875533_162.1 2021-09-02T08:03:42.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:42.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:42.616 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:03:42.664 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:03:42.665 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:03:42.862 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:03:42.862 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:03:42.862 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:03:42.862 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:03:42.863 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:03:42.863 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:03:42.863 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:42.863 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:42.864 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:03:42.864 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:42.864 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:03:42.864 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:42.865 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:42.865 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:03:42.865 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:42.865 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:03:42.866 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:42.866 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:03:42.866 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:42.866 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:42.867 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:42.867 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:42.867 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:03:42.867 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:42.867 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:42.868 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T08:03:42.868 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:42.868 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:42.868 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:42.869 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T08:03:42.869 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:42.869 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:42.869 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:42.870 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:42.870 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:03:42.870 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:42.870 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:42.871 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:42.871 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:42.871 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:42.871 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:42.871 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:42.872 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:42.872 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:42.872 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:42.872 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:42.873 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:03:42.873 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:42.873 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:42.873 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:42.874 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:42.874 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:42.874 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:42.874 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:42.875 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:42.875 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:42.875 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:42.875 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:03:42.875 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:42.876 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:03:42.876 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:03:42.876 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:03:42.876 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:42.877 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:03:42.878 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.875533_162.1 2021-09-02T08:03:47.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:47.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:47.881 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:03:47.931 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:03:47.931 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:03:48.012 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:03:48.012 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:03:48.013 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:03:48.013 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:03:48.013 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:03:48.013 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:03:48.014 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:48.014 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:48.014 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:03:48.014 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:48.014 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:03:48.015 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:48.015 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:48.015 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:03:48.015 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:48.016 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:03:48.016 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:48.016 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:03:48.016 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:48.017 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:48.017 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:48.017 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:48.017 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:03:48.017 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:48.018 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:48.018 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.929348_167.1", 2021-09-02T08:03:48.018 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:48.018 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:48.019 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:48.019 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.929348Z", 2021-09-02T08:03:48.019 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:48.019 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:48.020 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:48.020 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:48.020 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:03:48.020 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:48.021 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:48.025 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:48.025 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:48.025 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:48.026 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:48.027 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:48.028 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:48.028 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:48.028 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:48.028 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:48.029 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:03:48.029 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:48.029 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:48.030 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:48.030 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:48.030 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:48.030 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:48.031 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:48.031 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:48.031 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:48.031 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:48.031 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:03:48.032 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:48.032 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:03:48.032 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:03:48.033 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:03:48.033 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:48.033 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:03:48.034 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.929348_167.1 2021-09-02T08:03:53.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:53.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:53.037 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:03:53.081 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:03:53.081 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:03:53.156 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:03:53.156 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:03:53.157 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:03:53.157 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:03:53.157 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:03:53.158 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:03:53.158 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:53.158 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:53.158 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:03:53.159 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:53.159 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:03:53.159 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:53.159 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:53.159 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:03:53.160 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:53.160 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:03:53.160 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:53.160 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:03:53.161 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:53.161 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:53.161 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:53.161 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:53.162 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:03:53.162 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:53.162 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:53.162 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T08:03:53.163 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:53.163 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:53.163 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:53.163 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T08:03:53.164 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:53.164 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:53.164 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:53.164 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:53.164 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:03:53.165 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:53.165 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:53.165 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:53.165 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:53.166 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:53.166 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:53.166 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:53.166 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:53.166 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:53.167 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:53.167 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:53.167 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:03:53.167 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:53.168 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:53.168 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:53.168 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:53.168 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:53.168 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:53.169 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:53.169 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:53.169 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:53.169 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:53.170 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:03:53.170 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:53.170 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:03:53.170 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:03:53.171 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:03:53.171 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:53.171 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:03:53.172 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.910082_164.1 2021-09-02T08:03:58.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:58.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:03:58.176 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:03:58.247 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:03:58.248 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:03:58.703 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:03:58.704 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:03:58.704 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:03:58.704 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:03:58.705 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:03:58.705 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:03:58.705 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:58.705 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:58.706 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:03:58.706 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:58.706 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:03:58.707 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:58.707 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:58.707 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:03:58.708 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:58.708 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:03:58.708 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:58.708 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:03:58.709 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:58.709 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:58.709 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:58.709 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:58.710 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:03:58.710 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:58.710 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:58.711 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T08:03:58.711 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:58.711 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:58.711 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:58.712 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T08:03:58.712 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:58.712 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:58.712 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:58.713 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:58.713 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:03:58.713 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:58.713 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:58.713 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:58.714 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:58.714 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:58.714 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:58.714 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:58.715 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:58.715 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:58.715 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:58.715 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:03:58.716 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:03:58.716 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:03:58.716 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:03:58.716 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:03:58.717 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:03:58.717 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:03:58.717 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:03:58.718 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:03:58.718 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:03:58.718 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:58.718 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:58.719 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:03:58.719 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:03:58.719 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:03:58.719 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:03:58.720 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:03:58.720 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:03:58.720 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:03:58.721 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.910082_164.1 2021-09-02T08:04:03.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:03.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:03.724 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:04:03.828 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:04:03.828 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:04:04.154 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:04:04.155 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:04:04.155 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:04:04.155 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:04:04.155 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:04:04.156 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:04:04.156 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:04.156 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:04.156 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:04:04.157 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:04.157 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:04:04.157 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:04.157 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:04.158 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:04:04.158 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:04.158 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:04:04.158 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:04.159 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:04:04.159 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:04.159 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:04.159 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:04.160 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:04.160 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:04:04.160 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:04.160 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:04.160 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T08:04:04.161 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:04.161 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:04.161 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:04.161 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T08:04:04.162 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:04.162 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:04.162 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:04.162 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:04.163 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:04:04.163 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:04.163 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:04.163 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:04.163 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:04.164 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:04.164 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:04.164 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:04.164 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:04.166 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:04.166 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:04.166 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:04.166 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:04:04.167 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:04.167 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:04.167 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:04.167 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:04.168 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:04.168 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:04.168 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:04.168 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:04.169 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:04.169 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:04.169 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:04:04.169 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:04.169 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:04:04.170 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:04:04.170 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:04:04.170 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:04.170 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:04:04.176 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.910082_164.1 2021-09-02T08:04:09.177 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:09.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:09.179 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:04:09.268 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:04:09.268 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:04:09.759 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:04:09.759 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:04:09.760 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:04:09.760 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:04:09.760 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:04:09.760 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:04:09.760 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:09.761 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:09.761 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:04:09.761 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:09.761 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:04:09.762 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:09.762 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:09.762 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:04:09.762 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:09.763 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:04:09.763 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:09.763 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:04:09.763 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:09.764 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:09.764 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:09.764 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:09.764 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:04:09.764 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:09.765 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:09.765 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T08:04:09.765 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:09.765 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:09.766 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:09.766 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T08:04:09.766 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:09.766 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:09.767 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:09.767 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:09.767 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:04:09.768 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:09.768 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:09.768 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:09.768 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:09.769 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:09.769 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:09.769 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:09.769 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:09.770 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:09.770 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:09.770 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:09.770 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:04:09.771 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:09.771 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:09.771 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:09.771 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:09.771 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:09.772 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:09.772 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:09.772 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:09.772 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:09.773 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:09.773 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:04:09.773 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:09.773 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:04:09.773 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:04:09.774 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:04:09.774 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:09.774 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:04:09.776 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.103991_175.1 2021-09-02T08:04:14.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:14.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:14.778 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:04:14.822 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:04:14.823 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:04:14.891 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:04:14.891 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:04:14.892 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:04:14.892 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:04:14.892 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:04:14.892 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:04:14.893 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:14.893 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:14.893 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:04:14.893 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:14.894 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:04:14.894 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:14.894 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:14.894 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:04:14.895 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:14.895 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:04:14.895 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:14.896 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:04:14.897 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:14.897 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:14.897 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:14.898 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:14.898 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:04:14.898 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:14.898 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:14.898 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568129.951314_211.1", 2021-09-02T08:04:14.899 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:14.899 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:14.899 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:14.899 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:29.951314Z", 2021-09-02T08:04:14.900 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:14.900 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:14.900 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:14.900 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:14.901 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:04:14.901 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:14.901 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:14.901 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:14.902 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:14.902 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:14.902 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:14.902 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:14.902 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:14.903 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:14.903 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:14.903 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:14.903 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:04:14.903 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:14.904 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:14.904 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:14.904 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:14.904 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:14.905 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:14.905 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:14.905 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:14.905 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:14.905 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:14.906 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:04:14.906 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:14.906 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:04:14.906 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:04:14.907 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:04:14.907 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:14.907 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:04:14.908 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568129.951314_211.1 2021-09-02T08:04:19.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:19.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:19.912 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:04:19.954 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:04:19.955 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:04:20.029 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:04:20.029 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:04:20.029 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:04:20.030 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:04:20.030 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:04:20.030 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:04:20.030 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:20.031 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:20.031 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:04:20.031 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:20.031 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:04:20.032 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:20.032 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:20.032 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:04:20.032 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:20.033 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:04:20.033 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:20.033 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:04:20.033 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:20.034 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:20.034 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:20.034 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:20.034 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:04:20.034 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:20.035 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:20.035 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568129.951314_211.1", 2021-09-02T08:04:20.035 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:20.035 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:20.036 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:20.036 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:29.951314Z", 2021-09-02T08:04:20.036 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:20.036 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:20.036 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:20.037 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:20.037 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:04:20.037 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:20.037 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:20.038 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:20.038 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:20.038 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:20.038 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:20.039 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:20.039 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:20.039 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:20.039 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:20.040 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:20.040 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:04:20.040 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:20.040 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:20.040 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:20.041 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:20.041 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:20.041 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:20.041 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:20.042 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:20.042 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:20.042 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:20.042 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:04:20.043 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:20.043 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:04:20.043 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:04:20.043 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:04:20.043 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:20.044 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:04:20.045 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568129.951314_211.1 2021-09-02T08:04:25.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:25.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:25.048 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:04:25.091 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:04:25.092 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:04:25.160 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:04:25.160 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:04:25.160 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:04:25.161 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:04:25.161 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:04:25.161 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:04:25.161 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:25.161 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:25.162 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:04:25.162 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:25.162 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:04:25.162 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:25.163 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:25.163 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:04:25.163 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:25.163 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:04:25.164 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:25.164 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:04:25.164 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:25.164 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:25.165 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:25.165 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:25.165 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:04:25.165 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:25.166 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:25.166 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.985199_169.1", 2021-09-02T08:04:25.166 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:25.167 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:25.167 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:25.167 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.985199Z", 2021-09-02T08:04:25.167 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:25.168 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:25.168 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:25.168 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:25.168 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:04:25.169 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:25.169 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:25.169 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:25.169 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:25.169 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:25.170 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:25.170 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:25.170 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:25.170 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:25.171 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:25.171 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:25.171 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:04:25.171 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:25.172 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:25.172 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:25.172 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:25.172 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:25.173 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:25.173 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:25.173 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:25.173 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:25.173 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:25.174 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:04:25.174 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:25.174 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:04:25.175 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:04:25.175 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:04:25.175 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:25.175 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:04:25.177 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.985199_169.1 2021-09-02T08:04:30.178 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:30.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:30.180 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:04:30.236 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:04:30.237 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:04:30.467 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:04:30.467 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:04:30.468 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:04:30.468 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:04:30.468 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:04:30.468 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:04:30.469 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:30.469 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:30.469 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:04:30.469 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:30.469 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:04:30.470 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:30.470 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:30.470 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:04:30.471 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:30.471 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:04:30.471 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:30.471 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:04:30.472 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:30.472 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:30.472 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:30.473 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:30.473 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:04:30.473 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:30.473 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:30.474 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.929348_167.1", 2021-09-02T08:04:30.474 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:30.474 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:30.474 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:30.475 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.929348Z", 2021-09-02T08:04:30.475 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:30.475 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:30.475 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:30.475 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:30.476 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:04:30.476 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:30.476 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:30.476 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:30.477 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:30.477 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:30.477 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:30.477 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:30.478 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:30.478 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:30.478 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:30.478 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:30.478 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:04:30.479 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:30.479 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:30.479 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:30.479 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:30.480 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:30.480 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:30.480 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:30.480 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:30.481 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:30.481 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:30.481 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:04:30.481 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:30.481 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:04:30.482 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:04:30.482 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:04:30.482 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:30.482 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:04:30.483 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.929348_167.1 2021-09-02T08:04:35.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:35.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:35.486 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:04:35.571 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:04:35.573 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:04:35.862 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:04:35.863 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:04:35.863 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:04:35.863 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:04:35.863 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:04:35.864 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:04:35.864 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:35.864 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:35.864 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:04:35.864 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:35.865 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:04:35.865 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:35.865 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:35.865 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:04:35.866 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:35.866 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:04:35.866 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:35.866 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:04:35.867 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:35.867 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:35.867 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:35.867 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:35.868 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:04:35.868 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:35.868 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:35.868 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.990121_170.1", 2021-09-02T08:04:35.869 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:35.869 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:35.869 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:35.869 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.990121Z", 2021-09-02T08:04:35.870 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:35.870 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:35.870 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:35.870 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:35.870 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:04:35.871 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:35.871 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:35.871 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:35.872 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:35.872 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:35.872 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:35.873 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:35.873 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:35.873 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:35.873 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:35.874 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:35.874 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:04:35.874 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:35.874 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:35.874 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:35.875 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:35.875 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:35.875 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:35.875 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:35.876 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:35.876 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:35.876 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:35.876 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:04:35.876 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:35.877 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:04:35.877 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:04:35.877 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:04:35.877 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:35.878 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:04:35.879 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.990121_170.1 2021-09-02T08:04:40.880 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:40.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:40.882 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:04:40.954 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:04:40.955 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:04:41.556 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:04:41.556 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:04:41.556 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:04:41.557 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:04:41.557 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:04:41.557 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:04:41.557 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:41.558 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:41.558 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:04:41.558 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:41.558 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:04:41.559 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:41.559 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:41.559 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:04:41.559 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:41.559 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:04:41.560 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:41.560 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:04:41.560 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:41.560 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:41.561 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:41.561 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:41.561 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:04:41.561 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:41.561 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:41.562 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T08:04:41.562 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:41.562 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:41.562 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:41.563 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T08:04:41.563 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:41.563 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:41.563 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:41.563 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:41.564 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:04:41.564 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:41.564 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:41.564 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:41.564 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:41.565 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:41.565 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:41.565 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:41.565 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:41.566 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:41.566 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:41.566 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:41.566 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:04:41.567 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:41.567 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:41.567 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:41.567 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:41.568 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:41.568 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:41.568 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:41.568 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:41.568 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:41.569 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:41.569 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:04:41.569 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:41.569 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:04:41.570 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:04:41.570 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:04:41.570 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:41.570 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:04:41.571 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.910082_164.1 2021-09-02T08:04:46.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:46.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:46.574 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:04:46.651 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:04:46.651 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:04:47.269 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:04:47.269 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:04:47.269 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:04:47.269 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:04:47.270 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:04:47.270 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:04:47.270 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:47.270 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:47.271 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:04:47.271 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:47.271 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:04:47.271 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:47.272 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:47.272 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:04:47.272 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:47.272 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:04:47.273 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:47.273 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:04:47.273 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:47.273 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:47.273 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:47.274 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:47.274 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:04:47.274 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:47.274 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:47.275 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.043408_172.1", 2021-09-02T08:04:47.275 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:47.275 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:47.275 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:47.276 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.043408Z", 2021-09-02T08:04:47.276 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:47.276 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:47.276 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:47.276 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:47.277 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:04:47.277 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:47.277 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:47.277 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:47.278 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:47.278 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:47.278 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:47.278 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:47.278 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:47.279 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:47.279 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:47.279 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:47.279 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:04:47.280 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:47.280 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:47.280 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:47.280 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:47.280 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:47.281 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:47.281 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:47.281 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:47.281 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:47.282 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:47.282 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:04:47.282 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:47.282 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:04:47.283 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:04:47.283 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:04:47.283 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:47.283 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:04:47.284 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.043408_172.1 2021-09-02T08:04:52.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:52.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:52.286 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:04:52.335 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:04:52.335 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:04:52.593 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:04:52.593 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:04:52.593 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:04:52.594 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:04:52.594 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:04:52.594 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:04:52.594 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:52.595 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:52.595 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:04:52.595 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:52.595 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:04:52.596 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:52.596 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:52.596 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:04:52.597 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:52.597 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:04:52.597 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:52.597 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:04:52.598 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:52.598 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:52.598 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:52.598 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:52.599 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:04:52.599 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:52.599 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:52.599 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T08:04:52.600 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:52.600 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:52.600 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:52.600 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T08:04:52.600 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:52.601 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:52.601 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:52.601 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:52.601 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:04:52.602 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:52.602 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:52.602 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:52.602 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:52.603 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:52.603 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:52.603 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:52.603 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:52.603 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:52.604 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:52.604 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:52.604 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:04:52.604 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:52.605 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:52.605 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:52.605 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:52.605 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:52.606 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:52.606 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:52.606 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:52.606 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:52.606 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:52.607 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:04:52.607 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:52.607 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:04:52.607 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:04:52.608 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:04:52.608 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:52.608 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:04:52.609 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.103991_175.1 2021-09-02T08:04:57.611 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:57.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:04:57.613 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:04:57.662 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:04:57.663 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:04:57.736 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:04:57.736 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:04:57.737 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:04:57.737 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:04:57.738 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:04:57.738 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:04:57.739 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:57.739 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:57.739 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:04:57.740 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:57.740 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:04:57.741 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:57.741 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:57.742 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:04:57.742 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:57.743 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:04:57.743 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:57.744 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:04:57.744 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:57.744 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:57.745 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:57.745 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:57.746 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:04:57.746 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:57.747 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:57.747 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.103991_175.1", 2021-09-02T08:04:57.748 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:57.748 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:57.748 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:57.749 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.103991Z", 2021-09-02T08:04:57.749 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:57.750 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:57.750 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:57.751 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:57.751 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:04:57.752 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:57.752 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:57.752 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:57.753 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:57.753 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:57.754 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:57.754 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:57.755 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:57.755 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:57.756 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:57.756 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:04:57.757 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:04:57.757 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:04:57.757 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:04:57.758 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:04:57.758 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:04:57.759 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:04:57.759 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:04:57.760 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:04:57.760 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:04:57.760 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:57.761 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:57.761 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:04:57.762 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:04:57.762 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:04:57.762 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:04:57.763 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:04:57.763 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:04:57.764 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:04:57.766 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.103991_175.1 2021-09-02T08:05:02.767 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:02.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:02.768 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:05:02.811 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:05:02.811 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:05:02.893 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:05:02.893 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:05:02.893 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:05:02.894 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:05:02.894 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:05:02.894 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:05:02.894 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:02.894 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:02.895 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:05:02.895 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:02.895 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:05:02.895 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:02.896 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:02.896 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:05:02.896 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:02.896 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:05:02.896 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:02.897 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:05:02.897 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:02.897 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:02.897 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:02.898 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:02.898 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:05:02.898 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:02.898 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:02.898 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.990121_170.1", 2021-09-02T08:05:02.899 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:02.899 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:02.899 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:02.899 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.990121Z", 2021-09-02T08:05:02.900 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:02.900 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:02.900 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:02.900 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:02.900 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:05:02.901 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:02.901 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:02.901 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:02.901 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:02.901 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:02.902 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:02.902 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:02.902 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:02.902 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:02.903 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:02.903 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:02.903 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:05:02.903 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:02.903 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:02.904 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:02.904 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:02.904 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:02.904 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:02.904 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:02.905 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:02.905 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:02.905 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:02.905 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:05:02.906 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:02.906 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:05:02.906 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:05:02.906 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:05:02.906 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:02.907 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:05:02.908 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.990121_170.1 2021-09-02T08:05:07.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:07.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:07.910 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:05:07.998 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:05:07.998 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:05:08.339 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:05:08.339 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:05:08.339 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:05:08.340 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:05:08.340 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:05:08.340 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:05:08.340 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:08.340 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:08.341 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:05:08.341 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:08.341 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:05:08.341 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:08.342 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:08.342 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:05:08.342 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:08.342 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:05:08.343 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:08.343 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:05:08.343 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:08.343 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:08.344 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:08.344 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:08.344 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:05:08.344 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:08.344 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:08.345 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.043408_172.1", 2021-09-02T08:05:08.345 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:08.345 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:08.345 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:08.346 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.043408Z", 2021-09-02T08:05:08.346 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:08.346 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:08.346 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:08.346 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:08.347 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:05:08.347 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:08.347 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:08.347 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:08.348 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:08.348 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:08.348 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:08.348 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:08.349 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:08.349 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:08.349 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:08.349 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:08.349 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:05:08.350 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:08.350 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:08.350 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:08.350 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:08.351 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:08.351 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:08.351 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:08.351 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:08.351 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:08.352 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:08.352 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:05:08.352 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:08.352 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:05:08.353 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:05:08.353 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:05:08.353 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:08.353 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:05:08.355 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.043408_172.1 2021-09-02T08:05:13.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:13.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:13.358 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:05:13.407 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:05:13.407 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:05:13.537 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:05:13.538 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:05:13.538 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:05:13.539 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:05:13.539 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:05:13.539 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:05:13.539 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:13.539 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:13.540 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:05:13.540 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:13.540 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:05:13.540 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:13.541 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:13.541 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:05:13.541 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:13.541 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:05:13.541 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:13.542 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:05:13.542 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:13.542 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:13.542 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:13.543 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:13.543 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:05:13.543 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:13.543 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:13.543 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T08:05:13.544 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:13.544 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:13.544 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:13.544 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T08:05:13.545 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:13.545 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:13.545 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:13.545 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:13.546 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:05:13.546 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:13.546 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:13.546 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:13.546 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:13.547 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:13.547 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:13.547 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:13.547 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:13.548 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:13.548 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:13.548 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:13.548 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:05:13.549 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:13.549 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:13.549 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:13.549 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:13.549 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:13.550 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:13.550 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:13.550 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:13.550 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:13.551 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:13.551 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:05:13.551 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:13.551 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:05:13.551 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:05:13.552 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:05:13.552 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:13.552 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:05:13.553 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.875533_162.1 2021-09-02T08:05:18.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:18.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:18.556 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:05:18.617 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:05:18.617 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:05:19.128 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:05:19.128 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:05:19.128 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:05:19.129 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:05:19.129 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:05:19.129 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:05:19.129 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:19.130 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:19.130 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:05:19.130 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:19.131 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:05:19.131 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:19.131 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:19.131 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:05:19.132 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:19.132 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:05:19.132 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:19.132 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:05:19.132 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:19.133 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:19.133 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:19.133 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:19.133 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:05:19.134 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:19.134 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:19.134 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T08:05:19.134 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:19.135 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:19.135 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:19.135 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T08:05:19.135 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:19.136 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:19.136 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:19.136 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:19.137 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:05:19.137 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:19.137 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:19.137 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:19.137 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:19.138 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:19.138 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:19.138 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:19.138 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:19.139 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:19.139 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:19.139 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:19.139 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:05:19.140 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:19.140 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:19.140 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:19.140 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:19.140 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:19.141 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:19.141 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:19.141 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:19.141 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:19.142 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:19.142 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:05:19.142 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:19.142 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:05:19.143 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:05:19.143 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:05:19.143 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:19.143 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:05:19.144 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.910082_164.1 2021-09-02T08:05:24.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:24.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:24.148 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:05:24.218 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:05:24.219 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:05:24.817 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:05:24.817 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:05:24.818 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:05:24.818 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:05:24.818 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:05:24.818 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:05:24.819 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:24.819 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:24.819 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:05:24.819 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:24.820 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:05:24.820 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:24.820 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:24.820 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:05:24.821 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:24.821 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:05:24.821 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:24.821 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:05:24.822 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:24.822 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:24.822 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:24.822 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:24.823 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:05:24.823 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:24.823 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:24.823 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.099221_173.1", 2021-09-02T08:05:24.824 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:24.824 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:24.824 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:24.824 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.099221Z", 2021-09-02T08:05:24.825 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:24.825 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:24.825 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:24.825 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:24.826 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:05:24.826 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:24.826 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:24.827 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:24.827 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:24.827 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:24.827 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:24.827 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:24.828 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:24.828 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:24.828 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:24.828 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:24.829 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:05:24.829 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:24.829 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:24.830 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:24.830 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:24.830 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:24.830 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:24.831 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:24.831 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:24.831 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:24.831 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:24.831 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:05:24.832 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:24.832 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:05:24.832 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:05:24.832 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:05:24.833 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:24.833 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:05:24.833 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.099221_173.1 2021-09-02T08:05:29.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:29.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:29.836 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:05:29.920 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:05:29.920 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:05:30.354 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:05:30.354 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:05:30.355 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:05:30.355 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:05:30.355 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:05:30.355 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:05:30.356 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:30.356 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:30.356 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:05:30.356 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:30.357 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:05:30.357 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:30.357 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:30.357 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:05:30.357 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:30.358 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:05:30.358 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:30.358 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:05:30.358 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:30.359 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:30.359 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:30.359 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:30.359 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:05:30.360 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:30.360 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:30.360 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T08:05:30.360 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:30.361 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:30.361 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:30.361 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T08:05:30.361 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:30.361 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:30.362 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:30.362 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:30.362 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:05:30.362 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:30.363 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:30.363 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:30.363 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:30.363 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:30.363 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:30.364 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:30.364 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:30.364 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:30.364 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:30.364 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:30.365 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:05:30.365 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:30.365 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:30.365 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:30.366 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:30.366 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:30.366 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:30.366 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:30.367 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:30.367 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:30.367 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:30.367 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:05:30.367 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:30.368 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:05:30.368 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:05:30.368 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:05:30.368 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:30.369 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:05:30.370 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.906521_163.1 2021-09-02T08:05:35.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:35.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:35.373 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:05:35.457 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:05:35.458 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:05:35.994 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:05:35.995 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:05:35.995 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:05:35.995 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:05:35.995 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:05:35.996 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:05:35.996 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:35.996 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:35.996 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:05:35.997 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:35.997 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:05:35.997 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:35.997 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:35.997 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:05:35.998 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:35.998 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:05:35.998 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:35.998 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:05:35.999 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:35.999 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:35.999 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:35.999 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:36.000 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:05:36.000 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:36.000 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:36.000 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.929348_167.1", 2021-09-02T08:05:36.001 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:36.001 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:36.001 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:36.001 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.929348Z", 2021-09-02T08:05:36.002 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:36.002 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:36.002 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:36.002 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:36.002 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:05:36.003 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:36.003 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:36.003 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:36.003 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:36.004 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:36.004 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:36.004 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:36.004 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:36.005 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:36.005 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:36.005 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:36.005 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:05:36.005 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:36.006 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:36.006 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:36.006 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:36.006 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:36.007 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:36.007 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:36.007 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:36.007 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:36.007 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:36.008 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:05:36.008 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:36.008 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:05:36.008 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:05:36.009 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:05:36.009 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:36.009 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:05:36.011 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.929348_167.1 2021-09-02T08:05:41.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:41.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:41.014 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:05:41.087 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:05:41.087 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:05:41.721 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:05:41.721 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:05:41.722 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:05:41.722 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:05:41.722 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:05:41.722 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:05:41.723 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:41.723 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:41.723 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:05:41.723 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:41.723 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:05:41.724 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:41.724 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:41.724 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:05:41.724 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:41.725 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:05:41.725 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:41.725 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:05:41.725 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:41.725 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:41.726 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:41.726 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:41.726 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:05:41.727 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:41.727 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:41.727 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.985199_169.1", 2021-09-02T08:05:41.727 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:41.727 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:41.728 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:41.728 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.985199Z", 2021-09-02T08:05:41.728 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:41.728 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:41.729 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:41.729 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:41.729 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:05:41.729 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:41.729 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:41.730 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:41.730 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:41.730 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:41.730 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:41.731 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:41.731 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:41.731 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:41.731 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:41.731 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:41.732 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:05:41.732 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:41.732 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:41.732 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:41.733 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:41.733 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:41.733 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:41.733 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:41.733 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:41.734 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:41.734 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:41.734 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:05:41.734 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:41.735 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:05:41.735 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:05:41.735 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:05:41.736 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:41.736 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:05:41.736 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.985199_169.1 2021-09-02T08:05:46.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:46.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:46.739 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:05:46.795 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:05:46.795 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:05:47.301 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:05:47.301 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:05:47.302 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:05:47.302 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:05:47.302 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:05:47.302 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:05:47.302 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:47.303 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:47.303 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:05:47.303 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:47.303 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:05:47.304 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:47.304 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:47.304 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:05:47.305 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:47.305 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:05:47.305 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:47.305 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:05:47.305 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:47.306 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:47.306 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:47.306 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:47.306 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:05:47.307 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:47.307 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:47.308 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.911309_165.1", 2021-09-02T08:05:47.308 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:47.308 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:47.308 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:47.308 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.911309Z", 2021-09-02T08:05:47.309 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:47.309 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:47.309 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:47.309 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:47.310 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:05:47.310 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:47.310 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:47.311 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:47.311 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:47.311 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:47.311 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:47.312 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:47.312 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:47.312 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:47.312 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:47.312 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:47.313 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:05:47.313 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:47.313 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:47.313 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:47.314 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:47.314 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:47.314 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:47.314 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:47.315 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:47.315 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:47.315 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:47.315 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:05:47.316 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:47.316 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:05:47.316 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:05:47.316 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:05:47.317 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:47.317 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:05:47.317 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.911309_165.1 2021-09-02T08:05:52.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:52.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:52.320 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:05:52.369 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:05:52.370 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:05:52.450 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:05:52.451 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:05:52.451 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:05:52.451 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:05:52.451 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:05:52.452 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:05:52.452 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:52.452 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:52.452 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:05:52.453 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:52.453 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:05:52.453 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:52.453 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:52.453 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:05:52.454 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:52.454 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:05:52.454 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:52.454 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:05:52.455 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:52.455 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:52.455 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:52.455 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:52.455 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:05:52.456 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:52.456 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:52.456 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.099221_173.1", 2021-09-02T08:05:52.456 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:52.457 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:52.457 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:52.457 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.099221Z", 2021-09-02T08:05:52.457 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:52.457 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:52.458 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:52.458 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:52.458 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:05:52.458 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:52.459 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:52.459 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:52.459 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:52.459 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:52.459 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:52.460 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:52.460 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:52.460 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:52.460 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:52.460 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:52.461 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:05:52.461 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:52.461 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:52.461 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:52.462 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:52.462 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:52.462 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:52.462 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:52.462 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:52.463 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:52.463 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:52.463 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:05:52.463 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:52.464 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:05:52.464 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:05:52.464 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:05:52.464 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:52.464 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:05:52.466 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.099221_173.1 2021-09-02T08:05:57.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:57.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:05:57.469 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:05:57.518 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:05:57.518 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:05:57.596 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:05:57.597 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:05:57.597 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:05:57.597 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:05:57.598 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:05:57.598 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:05:57.598 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:57.598 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:57.599 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:05:57.599 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:57.599 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:05:57.599 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:57.600 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:57.600 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:05:57.600 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:57.600 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:05:57.601 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:57.601 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:05:57.601 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:57.601 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:57.602 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:57.602 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:57.602 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:05:57.602 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:57.602 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:57.603 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.914475_166.1", 2021-09-02T08:05:57.603 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:57.603 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:57.603 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:57.604 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.914475Z", 2021-09-02T08:05:57.604 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:57.604 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:57.604 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:57.604 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:57.605 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:05:57.605 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:57.605 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:57.605 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:57.606 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:57.606 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:57.606 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:57.606 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:57.607 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:57.607 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:57.607 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:57.607 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:05:57.607 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:05:57.608 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:05:57.608 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:05:57.608 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:05:57.608 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:05:57.608 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:05:57.609 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:05:57.609 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:05:57.609 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:05:57.609 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:57.610 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:57.610 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:05:57.610 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:05:57.610 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:05:57.610 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:05:57.611 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:05:57.611 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:05:57.611 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:05:57.612 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.914475_166.1 2021-09-02T08:06:02.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:06:02.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:06:02.615 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:06:02.659 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:06:02.659 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:06:02.733 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:06:02.734 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:06:02.734 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:06:02.734 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:06:02.734 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:06:02.735 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:06:02.735 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:02.735 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:02.735 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:06:02.736 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:02.736 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:06:02.736 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:02.736 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:02.736 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:06:02.737 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:02.737 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:06:02.737 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:02.737 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:06:02.738 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:02.738 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:02.738 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:02.738 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:02.738 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:06:02.739 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:02.739 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:02.739 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.039849_171.1", 2021-09-02T08:06:02.739 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:02.740 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:02.740 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:02.740 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.039849Z", 2021-09-02T08:06:02.740 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:02.741 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:02.741 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:02.741 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:02.741 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:06:02.741 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:02.742 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:02.742 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:06:02.742 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:02.742 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:02.743 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:02.743 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:06:02.743 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:02.743 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:02.743 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:02.744 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:02.744 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:06:02.744 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:02.744 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:02.745 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:06:02.745 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:02.745 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:02.745 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:02.746 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:06:02.746 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:02.746 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:02.746 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:02.746 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:06:02.747 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:02.747 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:06:02.747 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:06:02.747 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:06:02.748 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:02.748 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:06:02.749 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.039849_171.1 2021-09-02T08:06:07.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:06:07.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:06:07.752 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:06:07.844 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:06:07.845 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:06:08.192 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:06:08.193 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:06:08.193 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:06:08.193 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:06:08.193 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:06:08.194 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:06:08.194 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:08.194 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:08.194 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:06:08.194 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:08.195 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:06:08.195 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:08.195 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:08.195 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:06:08.196 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:08.196 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:06:08.196 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:08.196 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:06:08.197 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:08.197 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:08.197 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:08.197 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:08.198 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:06:08.198 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:08.198 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:08.198 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.127851_178.1", 2021-09-02T08:06:08.199 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:08.199 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:08.199 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:08.199 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.127851Z", 2021-09-02T08:06:08.200 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:08.200 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:08.200 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:08.200 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:08.201 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:06:08.201 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:08.201 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:08.201 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:06:08.202 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:08.202 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:08.202 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:08.202 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:06:08.203 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:08.203 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:08.203 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:08.203 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:08.203 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:06:08.204 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:08.204 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:08.204 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:06:08.204 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:08.205 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:08.205 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:08.205 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:06:08.205 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:08.205 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:08.206 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:08.206 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:06:08.206 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:08.206 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:06:08.207 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:06:08.207 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:06:08.207 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:08.207 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:06:08.208 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.127851_178.1 2021-09-02T08:06:13.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:06:13.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:06:13.211 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:06:13.267 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:06:13.267 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:06:13.351 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:06:13.352 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:06:13.352 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:06:13.352 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:06:13.352 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:06:13.353 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:06:13.353 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:13.353 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:13.353 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:06:13.354 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:13.354 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:06:13.354 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:13.354 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:13.355 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:06:13.355 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:13.355 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:06:13.355 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:13.356 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:06:13.356 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:13.356 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:13.356 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:13.356 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:13.357 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:06:13.357 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:13.357 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:13.357 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T08:06:13.358 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:13.358 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:13.358 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:13.358 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T08:06:13.359 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:13.359 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:13.359 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:13.359 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:13.359 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:06:13.360 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:13.360 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:13.360 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:06:13.361 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:13.361 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:13.361 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:13.361 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:06:13.362 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:13.362 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:13.362 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:13.362 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:13.363 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:06:13.363 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:13.363 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:13.363 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:06:13.364 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:13.364 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:13.364 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:13.364 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:06:13.364 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:13.365 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:13.365 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:13.365 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:06:13.365 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:13.366 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:06:13.366 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:06:13.366 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:06:13.366 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:13.366 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:06:13.368 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.906521_163.1 2021-09-02T08:06:18.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:06:18.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:06:18.372 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:06:18.470 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:06:18.471 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:06:18.959 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:06:18.960 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:06:18.960 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:06:18.960 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:06:18.960 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:06:18.961 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:06:18.961 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:18.961 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:18.961 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:06:18.962 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:18.962 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:06:18.962 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:18.962 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:18.963 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:06:18.963 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:18.963 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:06:18.963 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:18.963 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:06:18.964 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:18.964 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:18.964 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:18.964 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:18.965 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:06:18.965 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:18.965 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:18.965 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.875533_162.1", 2021-09-02T08:06:18.965 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:18.966 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:18.966 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:18.966 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.875533Z", 2021-09-02T08:06:18.966 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:18.967 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:18.967 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:18.967 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:18.967 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:06:18.967 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:18.968 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:18.968 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:06:18.968 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:18.968 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:18.969 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:18.969 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:06:18.969 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:18.969 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:18.969 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:18.970 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:18.970 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:06:18.970 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:18.970 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:18.971 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:06:18.971 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:18.971 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:18.971 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:18.971 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:06:18.972 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:18.972 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:18.972 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:18.972 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:06:18.972 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:18.973 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:06:18.973 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:06:18.973 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:06:18.974 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:18.974 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:06:18.974 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.875533_162.1 2021-09-02T08:06:23.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:06:23.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:06:23.977 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:06:24.060 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:06:24.060 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:06:24.505 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:06:24.505 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:06:24.506 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:06:24.506 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:06:24.506 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:06:24.506 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:06:24.507 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:24.507 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:24.507 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:06:24.507 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:24.508 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:06:24.508 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:24.508 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:24.509 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:06:24.509 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:24.509 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:06:24.509 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:24.510 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:06:24.510 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:24.510 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:24.510 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:24.511 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:24.511 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:06:24.511 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:24.511 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:24.511 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.911309_165.1", 2021-09-02T08:06:24.512 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:24.512 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:24.512 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:24.512 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.911309Z", 2021-09-02T08:06:24.513 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:24.513 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:24.513 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:24.514 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:24.514 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:06:24.514 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:24.514 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:24.514 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:06:24.516 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:24.516 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:24.516 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:24.516 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:06:24.517 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:24.517 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:24.517 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:24.517 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:24.518 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:06:24.518 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:24.518 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:24.518 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:06:24.519 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:24.519 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:24.519 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:24.519 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:06:24.520 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:24.520 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:24.521 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:24.521 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:06:24.522 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:24.522 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:06:24.522 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:06:24.522 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:06:24.523 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:24.523 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:06:24.524 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.911309_165.1 2021-09-02T08:06:29.526 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:06:29.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:06:29.527 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:06:29.575 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:06:29.575 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:06:29.684 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:06:29.684 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:06:29.684 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:06:29.685 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:06:29.685 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:06:29.685 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:06:29.685 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:29.686 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:29.686 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:06:29.686 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:29.686 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:06:29.687 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:29.687 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:29.687 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:06:29.687 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:29.687 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:06:29.688 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:29.688 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:06:29.688 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:29.688 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:29.689 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:29.689 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:29.689 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:06:29.689 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:29.689 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:29.690 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.906521_163.1", 2021-09-02T08:06:29.690 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:29.690 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:29.691 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:29.691 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.906521Z", 2021-09-02T08:06:29.691 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:29.691 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:29.691 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:29.692 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:29.692 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:06:29.692 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:29.692 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:29.693 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:06:29.693 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:29.693 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:29.693 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:29.693 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:06:29.694 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:29.694 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:29.694 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:29.694 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:29.695 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:06:29.695 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:29.695 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:29.695 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:06:29.696 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:29.696 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:29.696 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:29.696 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:06:29.696 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:29.697 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:29.697 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:29.697 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:06:29.697 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:29.698 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:06:29.698 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:06:29.698 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:06:29.698 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:29.699 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:06:29.700 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.906521_163.1 2021-09-02T08:06:34.701 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:06:34.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:06:34.703 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:06:34.749 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:06:34.749 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:06:34.819 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:06:34.820 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:06:34.820 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:06:34.820 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:06:34.820 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:06:34.821 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:06:34.821 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:34.821 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:34.821 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:06:34.822 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:34.822 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:06:34.822 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:34.822 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:34.823 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:06:34.823 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:34.823 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:06:34.823 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:34.824 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:06:34.824 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:34.824 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:34.824 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:34.825 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:34.825 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:06:34.825 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:34.825 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:34.826 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568126.910082_164.1", 2021-09-02T08:06:34.826 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:34.826 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:34.827 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:34.827 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:26.910082Z", 2021-09-02T08:06:34.827 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:34.827 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:34.828 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:34.828 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:34.828 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:06:34.829 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:34.829 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:34.829 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:06:34.830 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:34.830 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:34.830 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:34.830 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:06:34.831 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:34.831 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:34.831 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:34.831 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:34.831 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:06:34.832 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:34.832 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:34.832 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:06:34.833 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:34.833 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:34.833 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:34.833 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:06:34.834 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:34.834 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:34.834 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:34.834 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:06:34.835 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:34.835 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:06:34.835 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:06:34.836 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:06:34.836 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:34.836 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:06:34.838 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568126.910082_164.1 2021-09-02T08:06:39.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:06:39.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:06:39.841 DEBUG:teuthology.orchestra.run.smithi197:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:06:39.886 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:06:39.886 INFO:teuthology.orchestra.run.smithi197.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:06:39.954 INFO:teuthology.orchestra.run.smithi197.stdout:{ 2021-09-02T08:06:39.954 INFO:teuthology.orchestra.run.smithi197.stdout: "sync_status": { 2021-09-02T08:06:39.954 INFO:teuthology.orchestra.run.smithi197.stdout: "info": { 2021-09-02T08:06:39.954 INFO:teuthology.orchestra.run.smithi197.stdout: "status": "sync", 2021-09-02T08:06:39.955 INFO:teuthology.orchestra.run.smithi197.stdout: "num_shards": 4, 2021-09-02T08:06:39.955 INFO:teuthology.orchestra.run.smithi197.stdout: "period": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:06:39.955 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:39.956 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:39.956 INFO:teuthology.orchestra.run.smithi197.stdout: "markers": [ 2021-09-02T08:06:39.956 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:39.956 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 0, 2021-09-02T08:06:39.956 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:39.957 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:39.957 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630567930.734248_81.1", 2021-09-02T08:06:39.957 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:39.957 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 1, 2021-09-02T08:06:39.958 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:39.958 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:32:10.734248Z", 2021-09-02T08:06:39.958 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:39.958 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:39.959 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:39.959 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:39.959 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 1, 2021-09-02T08:06:39.959 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:39.960 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:39.960 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "1_1630568127.039849_171.1", 2021-09-02T08:06:39.960 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:39.961 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:39.961 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:39.961 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "2021-09-02T07:35:27.039849Z", 2021-09-02T08:06:39.962 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:39.962 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:39.962 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:39.962 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:39.963 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 2, 2021-09-02T08:06:39.963 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:39.963 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:39.963 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:06:39.964 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:39.964 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:39.964 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:39.964 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:06:39.965 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:39.965 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:39.965 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:39.965 INFO:teuthology.orchestra.run.smithi197.stdout: { 2021-09-02T08:06:39.966 INFO:teuthology.orchestra.run.smithi197.stdout: "key": 3, 2021-09-02T08:06:39.966 INFO:teuthology.orchestra.run.smithi197.stdout: "val": { 2021-09-02T08:06:39.966 INFO:teuthology.orchestra.run.smithi197.stdout: "state": 1, 2021-09-02T08:06:39.966 INFO:teuthology.orchestra.run.smithi197.stdout: "marker": "", 2021-09-02T08:06:39.967 INFO:teuthology.orchestra.run.smithi197.stdout: "next_step_marker": "", 2021-09-02T08:06:39.967 INFO:teuthology.orchestra.run.smithi197.stdout: "total_entries": 0, 2021-09-02T08:06:39.967 INFO:teuthology.orchestra.run.smithi197.stdout: "pos": 0, 2021-09-02T08:06:39.967 INFO:teuthology.orchestra.run.smithi197.stdout: "timestamp": "0.000000", 2021-09-02T08:06:39.968 INFO:teuthology.orchestra.run.smithi197.stdout: "realm_epoch": 2 2021-09-02T08:06:39.968 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:39.968 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:39.968 INFO:teuthology.orchestra.run.smithi197.stdout: ] 2021-09-02T08:06:39.969 INFO:teuthology.orchestra.run.smithi197.stdout: }, 2021-09-02T08:06:39.969 INFO:teuthology.orchestra.run.smithi197.stdout: "full_sync": { 2021-09-02T08:06:39.969 INFO:teuthology.orchestra.run.smithi197.stdout: "total": 1, 2021-09-02T08:06:39.970 INFO:teuthology.orchestra.run.smithi197.stdout: "complete": 1 2021-09-02T08:06:39.970 INFO:teuthology.orchestra.run.smithi197.stdout: } 2021-09-02T08:06:39.970 INFO:teuthology.orchestra.run.smithi197.stdout:} 2021-09-02T08:06:39.971 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1630569295.762188_292.1 target=1_1630568127.039849_171.1 2021-09-02T08:06:44.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... FAIL 2021-09-02T08:06:44.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T08:06:44.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T08:06:44.976 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-02T08:06:45.020 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:06:45.020 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:06:45.052 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T08:06:45.052 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6c53b8e1-6054-4a25-939b-3b4c792a06d9", 2021-09-02T08:06:45.053 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "remove_zg", 2021-09-02T08:06:45.053 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "remove_zg", 2021-09-02T08:06:45.053 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": "false", 2021-09-02T08:06:45.054 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [], 2021-09-02T08:06:45.054 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2021-09-02T08:06:45.054 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2021-09-02T08:06:45.054 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "", 2021-09-02T08:06:45.055 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [], 2021-09-02T08:06:45.055 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [], 2021-09-02T08:06:45.055 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "", 2021-09-02T08:06:45.055 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T08:06:45.056 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2021-09-02T08:06:45.056 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2021-09-02T08:06:45.056 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:06:45.056 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T08:06:45.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T08:06:45.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T08:06:45.059 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-02T08:06:45.138 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:06:45.138 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:06:45.173 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T08:06:45.174 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "530974b6-b9a1-4d27-994c-e1267e27b298", 2021-09-02T08:06:45.174 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "remove", 2021-09-02T08:06:45.174 INFO:teuthology.orchestra.run.smithi175.stdout: "domain_root": "remove.rgw.meta:root", 2021-09-02T08:06:45.175 INFO:teuthology.orchestra.run.smithi175.stdout: "control_pool": "remove.rgw.control", 2021-09-02T08:06:45.175 INFO:teuthology.orchestra.run.smithi175.stdout: "gc_pool": "remove.rgw.log:gc", 2021-09-02T08:06:45.175 INFO:teuthology.orchestra.run.smithi175.stdout: "lc_pool": "remove.rgw.log:lc", 2021-09-02T08:06:45.175 INFO:teuthology.orchestra.run.smithi175.stdout: "log_pool": "remove.rgw.log", 2021-09-02T08:06:45.176 INFO:teuthology.orchestra.run.smithi175.stdout: "intent_log_pool": "remove.rgw.log:intent", 2021-09-02T08:06:45.176 INFO:teuthology.orchestra.run.smithi175.stdout: "usage_log_pool": "remove.rgw.log:usage", 2021-09-02T08:06:45.176 INFO:teuthology.orchestra.run.smithi175.stdout: "roles_pool": "remove.rgw.meta:roles", 2021-09-02T08:06:45.177 INFO:teuthology.orchestra.run.smithi175.stdout: "reshard_pool": "remove.rgw.log:reshard", 2021-09-02T08:06:45.177 INFO:teuthology.orchestra.run.smithi175.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2021-09-02T08:06:45.177 INFO:teuthology.orchestra.run.smithi175.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2021-09-02T08:06:45.177 INFO:teuthology.orchestra.run.smithi175.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2021-09-02T08:06:45.177 INFO:teuthology.orchestra.run.smithi175.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2021-09-02T08:06:45.178 INFO:teuthology.orchestra.run.smithi175.stdout: "otp_pool": "remove.rgw.otp", 2021-09-02T08:06:45.178 INFO:teuthology.orchestra.run.smithi175.stdout: "system_key": { 2021-09-02T08:06:45.178 INFO:teuthology.orchestra.run.smithi175.stdout: "access_key": "", 2021-09-02T08:06:45.178 INFO:teuthology.orchestra.run.smithi175.stdout: "secret_key": "" 2021-09-02T08:06:45.179 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:45.179 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_pools": [ 2021-09-02T08:06:45.179 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:45.179 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "default-placement", 2021-09-02T08:06:45.180 INFO:teuthology.orchestra.run.smithi175.stdout: "val": { 2021-09-02T08:06:45.180 INFO:teuthology.orchestra.run.smithi175.stdout: "index_pool": "remove.rgw.buckets.index", 2021-09-02T08:06:45.180 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": { 2021-09-02T08:06:45.180 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD": { 2021-09-02T08:06:45.181 INFO:teuthology.orchestra.run.smithi175.stdout: "data_pool": "remove.rgw.buckets.data" 2021-09-02T08:06:45.181 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:06:45.182 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:45.182 INFO:teuthology.orchestra.run.smithi175.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2021-09-02T08:06:45.182 INFO:teuthology.orchestra.run.smithi175.stdout: "index_type": 0 2021-09-02T08:06:45.183 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:06:45.183 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:06:45.184 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:45.184 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T08:06:45.185 INFO:teuthology.orchestra.run.smithi175.stdout: "notif_pool": "remove.rgw.log:notif" 2021-09-02T08:06:45.185 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T08:06:45.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T08:06:45.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T08:06:45.188 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-02T08:06:45.254 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:06:45.255 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:06:45.332 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T08:06:45.332 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:06:45.333 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 6, 2021-09-02T08:06:45.333 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "dcf62b62-291a-46a6-867e-5263b7775f6a", 2021-09-02T08:06:45.333 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2021-09-02T08:06:45.333 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2021-09-02T08:06:45.334 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:06:45.334 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2021-09-02T08:06:45.334 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:45.334 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "6c53b8e1-6054-4a25-939b-3b4c792a06d9", 2021-09-02T08:06:45.335 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "remove_zg", 2021-09-02T08:06:45.335 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "remove_zg", 2021-09-02T08:06:45.335 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": "false", 2021-09-02T08:06:45.335 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [], 2021-09-02T08:06:45.336 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2021-09-02T08:06:45.336 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2021-09-02T08:06:45.336 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "530974b6-b9a1-4d27-994c-e1267e27b298", 2021-09-02T08:06:45.336 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2021-09-02T08:06:45.337 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:45.337 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "530974b6-b9a1-4d27-994c-e1267e27b298", 2021-09-02T08:06:45.337 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "remove", 2021-09-02T08:06:45.337 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [], 2021-09-02T08:06:45.337 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": "false", 2021-09-02T08:06:45.338 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": "false", 2021-09-02T08:06:45.338 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2021-09-02T08:06:45.338 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": "false", 2021-09-02T08:06:45.338 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2021-09-02T08:06:45.339 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": "true", 2021-09-02T08:06:45.339 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2021-09-02T08:06:45.339 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "" 2021-09-02T08:06:45.339 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:06:45.339 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:45.340 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2021-09-02T08:06:45.340 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:45.340 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2021-09-02T08:06:45.341 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2021-09-02T08:06:45.341 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2021-09-02T08:06:45.341 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2021-09-02T08:06:45.341 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T08:06:45.342 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:06:45.342 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:45.342 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2021-09-02T08:06:45.342 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T08:06:45.343 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2021-09-02T08:06:45.343 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2021-09-02T08:06:45.343 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:06:45.343 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:45.344 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:45.344 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2021-09-02T08:06:45.344 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2021-09-02T08:06:45.344 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2021-09-02T08:06:45.345 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": "true", 2021-09-02T08:06:45.345 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2021-09-02T08:06:45.345 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2021-09-02T08:06:45.345 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:45.345 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2021-09-02T08:06:45.346 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2021-09-02T08:06:45.346 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T08:06:45.346 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2021-09-02T08:06:45.346 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:45.347 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "5bdffc94-40ec-4bcf-a75a-819f7216b883", 2021-09-02T08:06:45.347 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2021-09-02T08:06:45.347 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2021-09-02T08:06:45.347 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2021-09-02T08:06:45.348 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:45.348 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": "false", 2021-09-02T08:06:45.349 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": "true", 2021-09-02T08:06:45.349 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2021-09-02T08:06:45.349 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": "false", 2021-09-02T08:06:45.351 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2021-09-02T08:06:45.351 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": "true", 2021-09-02T08:06:45.351 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2021-09-02T08:06:45.352 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "" 2021-09-02T08:06:45.352 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:45.352 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:45.352 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T08:06:45.353 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2021-09-02T08:06:45.353 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2021-09-02T08:06:45.353 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2021-09-02T08:06:45.353 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:45.353 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": "false", 2021-09-02T08:06:45.354 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": "true", 2021-09-02T08:06:45.354 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2021-09-02T08:06:45.354 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": "false", 2021-09-02T08:06:45.354 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2021-09-02T08:06:45.355 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": "true", 2021-09-02T08:06:45.355 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2021-09-02T08:06:45.355 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "" 2021-09-02T08:06:45.356 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:45.356 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:45.356 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "a0dd8689-bd06-4f6e-8710-337db82a68b6", 2021-09-02T08:06:45.357 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone4", 2021-09-02T08:06:45.357 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2021-09-02T08:06:45.357 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi197.front.sepia.ceph.com:8001" 2021-09-02T08:06:45.357 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:45.358 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": "false", 2021-09-02T08:06:45.358 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": "true", 2021-09-02T08:06:45.358 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2021-09-02T08:06:45.358 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": "false", 2021-09-02T08:06:45.359 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "pubsub", 2021-09-02T08:06:45.360 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": "false", 2021-09-02T08:06:45.360 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [ 2021-09-02T08:06:45.360 INFO:teuthology.orchestra.run.smithi175.stdout: "test-zone1" 2021-09-02T08:06:45.361 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:45.361 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "" 2021-09-02T08:06:45.361 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:45.361 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:45.362 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e43b32ce-d30a-498d-8211-34e9897b4509", 2021-09-02T08:06:45.362 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2021-09-02T08:06:45.362 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2021-09-02T08:06:45.362 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi197.front.sepia.ceph.com:8000" 2021-09-02T08:06:45.363 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:45.363 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": "false", 2021-09-02T08:06:45.363 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": "true", 2021-09-02T08:06:45.363 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2021-09-02T08:06:45.364 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": "false", 2021-09-02T08:06:45.364 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2021-09-02T08:06:45.364 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": "false", 2021-09-02T08:06:45.364 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2021-09-02T08:06:45.365 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "" 2021-09-02T08:06:45.365 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:06:45.365 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:45.365 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2021-09-02T08:06:45.365 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:45.366 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2021-09-02T08:06:45.366 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2021-09-02T08:06:45.366 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2021-09-02T08:06:45.366 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2021-09-02T08:06:45.367 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T08:06:45.367 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:06:45.367 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:45.367 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2021-09-02T08:06:45.368 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T08:06:45.368 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2021-09-02T08:06:45.368 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2021-09-02T08:06:45.368 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:06:45.368 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:06:45.369 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:45.369 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2021-09-02T08:06:45.369 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:45.369 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "530974b6-b9a1-4d27-994c-e1267e27b298", 2021-09-02T08:06:45.370 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 3416713252 2021-09-02T08:06:45.370 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:45.370 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:45.370 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "5bdffc94-40ec-4bcf-a75a-819f7216b883", 2021-09-02T08:06:45.371 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 3291489215 2021-09-02T08:06:45.371 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:45.371 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:45.371 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T08:06:45.371 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2303553998 2021-09-02T08:06:45.372 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:45.372 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:45.372 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "a0dd8689-bd06-4f6e-8710-337db82a68b6", 2021-09-02T08:06:45.372 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 3813003288 2021-09-02T08:06:45.373 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:45.373 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:45.373 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "e43b32ce-d30a-498d-8211-34e9897b4509", 2021-09-02T08:06:45.373 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 1397540283 2021-09-02T08:06:45.373 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:06:45.374 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T08:06:45.374 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:45.374 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2021-09-02T08:06:45.374 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T08:06:45.375 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2021-09-02T08:06:45.375 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2021-09-02T08:06:45.375 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2021-09-02T08:06:45.375 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2021-09-02T08:06:45.376 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2021-09-02T08:06:45.376 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2021-09-02T08:06:45.376 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2021-09-02T08:06:45.376 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:45.376 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2021-09-02T08:06:45.377 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2021-09-02T08:06:45.377 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2021-09-02T08:06:45.377 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2021-09-02T08:06:45.377 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2021-09-02T08:06:45.378 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2021-09-02T08:06:45.378 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:06:45.378 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:45.378 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T08:06:45.379 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2021-09-02T08:06:45.379 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T08:06:45.379 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T08:06:45.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T08:06:45.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T08:06:45.381 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-02T08:06:45.447 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:06:45.448 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:06:45.497 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T08:06:45.497+0000 7fde6fa40380 0 zone id 530974b6-b9a1-4d27-994c-e1267e27b298 is not a part of zonegroup test-zonegroup 2021-09-02T08:06:45.497 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T08:06:45.497+0000 7fde6fa40380 0 zone id 530974b6-b9a1-4d27-994c-e1267e27b298 is not a part of zonegroup default 2021-09-02T08:06:45.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T08:06:45.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T08:06:45.509 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-02T08:06:45.566 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:06:45.567 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:06:45.646 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T08:06:45.646+0000 7f8c0d372380 0 ERROR: zonegroup remove_zg has a non existent master zone 2021-09-02T08:06:45.653 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-09-02T08:06:45.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T08:06:45.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T08:06:45.655 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-02T08:06:45.721 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:06:45.722 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:06:45.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T08:06:45.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-02T08:06:45.805 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-02T08:06:45.868 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:06:45.869 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:06:46.011 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2021-09-02T08:06:46.011 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:06:46.012 INFO:teuthology.orchestra.run.smithi175.stdout: "epoch": 7, 2021-09-02T08:06:46.012 INFO:teuthology.orchestra.run.smithi175.stdout: "predecessor_uuid": "dcf62b62-291a-46a6-867e-5263b7775f6a", 2021-09-02T08:06:46.012 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_status": [], 2021-09-02T08:06:46.012 INFO:teuthology.orchestra.run.smithi175.stdout: "period_map": { 2021-09-02T08:06:46.013 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "3c6be1ae-90c2-49d2-9d54-568c49b66f2d", 2021-09-02T08:06:46.014 INFO:teuthology.orchestra.run.smithi175.stdout: "zonegroups": [ 2021-09-02T08:06:46.014 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:46.014 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "test-zonegroup", 2021-09-02T08:06:46.015 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zonegroup", 2021-09-02T08:06:46.015 INFO:teuthology.orchestra.run.smithi175.stdout: "api_name": "", 2021-09-02T08:06:46.015 INFO:teuthology.orchestra.run.smithi175.stdout: "is_master": "true", 2021-09-02T08:06:46.015 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2021-09-02T08:06:46.015 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2021-09-02T08:06:46.016 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:46.016 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames": [], 2021-09-02T08:06:46.016 INFO:teuthology.orchestra.run.smithi175.stdout: "hostnames_s3website": [], 2021-09-02T08:06:46.016 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T08:06:46.017 INFO:teuthology.orchestra.run.smithi175.stdout: "zones": [ 2021-09-02T08:06:46.017 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:46.017 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "5bdffc94-40ec-4bcf-a75a-819f7216b883", 2021-09-02T08:06:46.017 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone3", 2021-09-02T08:06:46.018 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2021-09-02T08:06:46.018 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8001" 2021-09-02T08:06:46.018 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:46.018 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": "false", 2021-09-02T08:06:46.019 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": "true", 2021-09-02T08:06:46.019 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2021-09-02T08:06:46.019 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": "false", 2021-09-02T08:06:46.019 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2021-09-02T08:06:46.019 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": "true", 2021-09-02T08:06:46.020 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2021-09-02T08:06:46.020 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "" 2021-09-02T08:06:46.020 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:46.020 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:46.021 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T08:06:46.021 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone1", 2021-09-02T08:06:46.021 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2021-09-02T08:06:46.021 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi175.front.sepia.ceph.com:8000" 2021-09-02T08:06:46.021 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:46.022 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": "false", 2021-09-02T08:06:46.022 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": "true", 2021-09-02T08:06:46.022 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2021-09-02T08:06:46.022 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": "false", 2021-09-02T08:06:46.023 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2021-09-02T08:06:46.023 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": "true", 2021-09-02T08:06:46.023 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2021-09-02T08:06:46.023 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "" 2021-09-02T08:06:46.024 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:46.024 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:46.024 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "a0dd8689-bd06-4f6e-8710-337db82a68b6", 2021-09-02T08:06:46.024 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone4", 2021-09-02T08:06:46.024 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2021-09-02T08:06:46.025 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi197.front.sepia.ceph.com:8001" 2021-09-02T08:06:46.025 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:46.025 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": "false", 2021-09-02T08:06:46.025 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": "true", 2021-09-02T08:06:46.026 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2021-09-02T08:06:46.026 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": "false", 2021-09-02T08:06:46.026 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "pubsub", 2021-09-02T08:06:46.026 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": "false", 2021-09-02T08:06:46.027 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [ 2021-09-02T08:06:46.027 INFO:teuthology.orchestra.run.smithi175.stdout: "test-zone1" 2021-09-02T08:06:46.027 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:46.027 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "" 2021-09-02T08:06:46.027 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:46.028 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:46.028 INFO:teuthology.orchestra.run.smithi175.stdout: "id": "e43b32ce-d30a-498d-8211-34e9897b4509", 2021-09-02T08:06:46.028 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "test-zone2", 2021-09-02T08:06:46.028 INFO:teuthology.orchestra.run.smithi175.stdout: "endpoints": [ 2021-09-02T08:06:46.028 INFO:teuthology.orchestra.run.smithi175.stdout: "http://smithi197.front.sepia.ceph.com:8000" 2021-09-02T08:06:46.029 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:46.029 INFO:teuthology.orchestra.run.smithi175.stdout: "log_meta": "false", 2021-09-02T08:06:46.029 INFO:teuthology.orchestra.run.smithi175.stdout: "log_data": "true", 2021-09-02T08:06:46.029 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_index_max_shards": 11, 2021-09-02T08:06:46.030 INFO:teuthology.orchestra.run.smithi175.stdout: "read_only": "false", 2021-09-02T08:06:46.030 INFO:teuthology.orchestra.run.smithi175.stdout: "tier_type": "", 2021-09-02T08:06:46.030 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from_all": "false", 2021-09-02T08:06:46.030 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_from": [], 2021-09-02T08:06:46.031 INFO:teuthology.orchestra.run.smithi175.stdout: "redirect_zone": "" 2021-09-02T08:06:46.031 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:06:46.031 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:46.031 INFO:teuthology.orchestra.run.smithi175.stdout: "placement_targets": [ 2021-09-02T08:06:46.031 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:46.032 INFO:teuthology.orchestra.run.smithi175.stdout: "name": "default-placement", 2021-09-02T08:06:46.032 INFO:teuthology.orchestra.run.smithi175.stdout: "tags": [], 2021-09-02T08:06:46.032 INFO:teuthology.orchestra.run.smithi175.stdout: "storage_classes": [ 2021-09-02T08:06:46.032 INFO:teuthology.orchestra.run.smithi175.stdout: "STANDARD" 2021-09-02T08:06:46.033 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T08:06:46.033 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:06:46.033 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:46.033 INFO:teuthology.orchestra.run.smithi175.stdout: "default_placement": "default-placement", 2021-09-02T08:06:46.033 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T08:06:46.034 INFO:teuthology.orchestra.run.smithi175.stdout: "sync_policy": { 2021-09-02T08:06:46.034 INFO:teuthology.orchestra.run.smithi175.stdout: "groups": [] 2021-09-02T08:06:46.034 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:06:46.034 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:06:46.035 INFO:teuthology.orchestra.run.smithi175.stdout: ], 2021-09-02T08:06:46.035 INFO:teuthology.orchestra.run.smithi175.stdout: "short_zone_ids": [ 2021-09-02T08:06:46.035 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:46.036 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "5bdffc94-40ec-4bcf-a75a-819f7216b883", 2021-09-02T08:06:46.037 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 3291489215 2021-09-02T08:06:46.037 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:46.037 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:46.037 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T08:06:46.038 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 2303553998 2021-09-02T08:06:46.038 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:46.038 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:46.038 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "a0dd8689-bd06-4f6e-8710-337db82a68b6", 2021-09-02T08:06:46.039 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 3813003288 2021-09-02T08:06:46.039 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:46.039 INFO:teuthology.orchestra.run.smithi175.stdout: { 2021-09-02T08:06:46.039 INFO:teuthology.orchestra.run.smithi175.stdout: "key": "e43b32ce-d30a-498d-8211-34e9897b4509", 2021-09-02T08:06:46.040 INFO:teuthology.orchestra.run.smithi175.stdout: "val": 1397540283 2021-09-02T08:06:46.040 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:06:46.040 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2021-09-02T08:06:46.040 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:46.041 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zonegroup": "test-zonegroup", 2021-09-02T08:06:46.041 INFO:teuthology.orchestra.run.smithi175.stdout: "master_zone": "8b6c886f-f9c4-4c5c-b256-edd3d1362f1c", 2021-09-02T08:06:46.041 INFO:teuthology.orchestra.run.smithi175.stdout: "period_config": { 2021-09-02T08:06:46.041 INFO:teuthology.orchestra.run.smithi175.stdout: "bucket_quota": { 2021-09-02T08:06:46.041 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2021-09-02T08:06:46.042 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2021-09-02T08:06:46.042 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2021-09-02T08:06:46.042 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2021-09-02T08:06:46.042 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2021-09-02T08:06:46.043 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:46.043 INFO:teuthology.orchestra.run.smithi175.stdout: "user_quota": { 2021-09-02T08:06:46.043 INFO:teuthology.orchestra.run.smithi175.stdout: "enabled": false, 2021-09-02T08:06:46.043 INFO:teuthology.orchestra.run.smithi175.stdout: "check_on_raw": false, 2021-09-02T08:06:46.043 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size": -1, 2021-09-02T08:06:46.044 INFO:teuthology.orchestra.run.smithi175.stdout: "max_size_kb": 0, 2021-09-02T08:06:46.044 INFO:teuthology.orchestra.run.smithi175.stdout: "max_objects": -1 2021-09-02T08:06:46.044 INFO:teuthology.orchestra.run.smithi175.stdout: } 2021-09-02T08:06:46.044 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2021-09-02T08:06:46.045 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_id": "7cd628c3-b272-4627-b6df-82e8f9eb7cb4", 2021-09-02T08:06:46.045 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_name": "test-realm", 2021-09-02T08:06:46.045 INFO:teuthology.orchestra.run.smithi175.stdout: "realm_epoch": 2 2021-09-02T08:06:46.045 INFO:teuthology.orchestra.run.smithi175.stdout:} 2021-09-02T08:06:46.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2021-09-02T08:06:46.047 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-46 2021-09-02T08:06:47.003 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=efybpk-47 2021-09-02T08:06:47.065 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=efybpk-48 2021-09-02T08:06:47.128 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-49 2021-09-02T08:06:47.130 INFO:tasks.rgw.c1.client.0.smithi175.stdout:*** Caught signal (Segmentation fault) ** 2021-09-02T08:06:47.130 INFO:tasks.rgw.c1.client.0.smithi175.stdout: in thread 7f9b04abc700 thread_name:radosgw 2021-09-02T08:06:47.130 INFO:tasks.rgw.c1.client.0.smithi175.stdout: ceph version 17.0.0-7401-g9366f375 (9366f375ac31713c35e559edbd160565b784db1d) quincy (dev) 2021-09-02T08:06:47.131 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f9ba0647b20] 2021-09-02T08:06:47.131 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x5be) [0x7f9ba325bf4e] 2021-09-02T08:06:47.131 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x185c) [0x7f9ba3279cac] 2021-09-02T08:06:47.131 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 4: /lib64/libradosgw.so.2(+0x54b3ed) [0x7f9ba31ad3ed] 2021-09-02T08:06:47.132 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 5: /lib64/libradosgw.so.2(+0x54c9d0) [0x7f9ba31ae9d0] 2021-09-02T08:06:47.132 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 6: /lib64/libradosgw.so.2(+0x54cb2c) [0x7f9ba31aeb2c] 2021-09-02T08:06:47.132 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 7: make_fcontext() 2021-09-02T08:06:47.133 INFO:tasks.rgw.c1.client.0.smithi175.stdout:2021-09-02T08:06:47.129+0000 7f9b04abc700 -1 *** Caught signal (Segmentation fault) ** 2021-09-02T08:06:47.133 INFO:tasks.rgw.c1.client.0.smithi175.stdout: in thread 7f9b04abc700 thread_name:radosgw 2021-09-02T08:06:47.133 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 2021-09-02T08:06:47.133 INFO:tasks.rgw.c1.client.0.smithi175.stdout: ceph version 17.0.0-7401-g9366f375 (9366f375ac31713c35e559edbd160565b784db1d) quincy (dev) 2021-09-02T08:06:47.133 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f9ba0647b20] 2021-09-02T08:06:47.134 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x5be) [0x7f9ba325bf4e] 2021-09-02T08:06:47.134 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x185c) [0x7f9ba3279cac] 2021-09-02T08:06:47.134 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 4: /lib64/libradosgw.so.2(+0x54b3ed) [0x7f9ba31ad3ed] 2021-09-02T08:06:47.134 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 5: /lib64/libradosgw.so.2(+0x54c9d0) [0x7f9ba31ae9d0] 2021-09-02T08:06:47.135 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 6: /lib64/libradosgw.so.2(+0x54cb2c) [0x7f9ba31aeb2c] 2021-09-02T08:06:47.135 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 7: make_fcontext() 2021-09-02T08:06:47.135 INFO:tasks.rgw.c1.client.0.smithi175.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-09-02T08:06:47.135 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 2021-09-02T08:06:47.148 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 0> 2021-09-02T08:06:47.129+0000 7f9b04abc700 -1 *** Caught signal (Segmentation fault) ** 2021-09-02T08:06:47.148 INFO:tasks.rgw.c1.client.0.smithi175.stdout: in thread 7f9b04abc700 thread_name:radosgw 2021-09-02T08:06:47.148 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 2021-09-02T08:06:47.149 INFO:tasks.rgw.c1.client.0.smithi175.stdout: ceph version 17.0.0-7401-g9366f375 (9366f375ac31713c35e559edbd160565b784db1d) quincy (dev) 2021-09-02T08:06:47.149 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f9ba0647b20] 2021-09-02T08:06:47.149 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x5be) [0x7f9ba325bf4e] 2021-09-02T08:06:47.149 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x185c) [0x7f9ba3279cac] 2021-09-02T08:06:47.150 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 4: /lib64/libradosgw.so.2(+0x54b3ed) [0x7f9ba31ad3ed] 2021-09-02T08:06:47.150 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 5: /lib64/libradosgw.so.2(+0x54c9d0) [0x7f9ba31ae9d0] 2021-09-02T08:06:47.150 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 6: /lib64/libradosgw.so.2(+0x54cb2c) [0x7f9ba31aeb2c] 2021-09-02T08:06:47.150 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 7: make_fcontext() 2021-09-02T08:06:47.151 INFO:tasks.rgw.c1.client.0.smithi175.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-09-02T08:06:47.151 INFO:tasks.rgw.c1.client.0.smithi175.stdout: 2021-09-02T08:06:47.232 INFO:tasks.rgw.c1.client.0.smithi175.stderr:daemon-helper: command crashed with signal 11 2021-09-02T08:06:52.381 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-09-02T08:06:57.987 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2021-09-02T08:07:03.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~11s 2021-09-02T08:07:09.199 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~17s 2021-09-02T08:07:14.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~22s 2021-09-02T08:07:20.411 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~28s 2021-09-02T08:07:26.019 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~34s 2021-09-02T08:07:31.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~39s 2021-09-02T08:07:33.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... rgw_multi.tests.test_set_bucket_policy ... ERROR 2021-09-02T08:07:33.233 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-50 2021-09-02T08:07:37.230 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~45s 2021-09-02T08:07:42.835 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~50s 2021-09-02T08:07:48.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~56s 2021-09-02T08:07:54.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~62s 2021-09-02T08:07:59.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~67s 2021-09-02T08:08:05.259 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~73s 2021-09-02T08:08:10.865 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~78s 2021-09-02T08:08:16.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~84s 2021-09-02T08:08:22.075 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~90s 2021-09-02T08:08:27.679 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~95s 2021-09-02T08:08:33.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~101s 2021-09-02T08:08:38.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~107s 2021-09-02T08:08:44.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~112s 2021-09-02T08:08:48.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ERROR 2021-09-02T08:08:48.200 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-51 2021-09-02T08:08:50.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~118s 2021-09-02T08:08:55.708 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~123s 2021-09-02T08:09:01.314 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~129s 2021-09-02T08:09:06.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~135s 2021-09-02T08:09:12.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~140s 2021-09-02T08:09:18.136 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~146s 2021-09-02T08:09:23.741 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~151s 2021-09-02T08:09:29.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~157s 2021-09-02T08:09:34.950 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~163s 2021-09-02T08:09:40.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~168s 2021-09-02T08:09:46.161 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~174s 2021-09-02T08:09:51.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~179s 2021-09-02T08:09:57.372 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~185s 2021-09-02T08:10:02.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~191s 2021-09-02T08:10:08.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~196s 2021-09-02T08:10:09.967 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ERROR 2021-09-02T08:10:09.968 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-52 2021-09-02T08:10:14.193 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~202s 2021-09-02T08:10:19.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~207s 2021-09-02T08:10:25.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~213s 2021-09-02T08:10:31.009 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~219s 2021-09-02T08:10:36.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~224s 2021-09-02T08:10:42.219 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~230s 2021-09-02T08:10:47.825 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~235s 2021-09-02T08:10:49.536 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ERROR 2021-09-02T08:10:49.537 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-53 2021-09-02T08:10:53.430 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~241s 2021-09-02T08:10:59.033 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~247s 2021-09-02T08:11:04.639 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~252s 2021-09-02T08:11:10.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~258s 2021-09-02T08:11:15.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~263s 2021-09-02T08:11:21.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~269s 2021-09-02T08:11:27.064 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~275s 2021-09-02T08:11:32.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~280s 2021-09-02T08:11:38.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~286s 2021-09-02T08:11:43.900 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~292s 2021-09-02T08:11:49.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~297s 2021-09-02T08:11:55.114 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~303s 2021-09-02T08:11:55.114 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2021-09-02T08:11:55.718 INFO:tasks.ceph.c1.osd.0:Sent signal 15 2021-09-02T08:11:55.720 INFO:tasks.ceph.c1.osd.1:Sent signal 15 2021-09-02T08:11:55.721 INFO:tasks.ceph.c1.osd.2:Sent signal 15 2021-09-02T08:11:55.721 INFO:tasks.ceph.c1.mon.a:Sent signal 15 2021-09-02T08:11:55.722 INFO:tasks.rgw.c1.client.1:Sent signal 15 2021-09-02T08:11:55.722 INFO:tasks.ceph.c1.mgr.x:Sent signal 15 2021-09-02T08:11:55.723 INFO:tasks.ceph.c1.osd.0.smithi175.stderr:2021-09-02T08:11:55.720+0000 7feb193a0700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 0 (PID: 30425) UID: 0 2021-09-02T08:11:55.724 INFO:tasks.ceph.c1.osd.0.smithi175.stderr:2021-09-02T08:11:55.720+0000 7feb193a0700 -1 osd.0 61 *** Got signal Terminated *** 2021-09-02T08:11:55.724 INFO:tasks.ceph.c1.osd.0.smithi175.stderr:2021-09-02T08:11:55.720+0000 7feb193a0700 -1 osd.0 61 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-09-02T08:11:55.725 INFO:tasks.ceph.c1.osd.1.smithi175.stderr:2021-09-02T08:11:55.723+0000 7fc2d38d6700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 1 (PID: 30429) UID: 0 2021-09-02T08:11:55.725 INFO:tasks.ceph.c1.osd.1.smithi175.stderr:2021-09-02T08:11:55.723+0000 7fc2d38d6700 -1 osd.1 61 *** Got signal Terminated *** 2021-09-02T08:11:55.725 INFO:tasks.ceph.c1.osd.1.smithi175.stderr:2021-09-02T08:11:55.723+0000 7fc2d38d6700 -1 osd.1 61 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-09-02T08:11:55.726 INFO:tasks.ceph.c1.osd.2.smithi175.stderr:2021-09-02T08:11:55.723+0000 7ff61ac88700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 2 (PID: 30431) UID: 0 2021-09-02T08:11:55.726 INFO:tasks.ceph.c1.osd.2.smithi175.stderr:2021-09-02T08:11:55.723+0000 7ff61ac88700 -1 osd.2 61 *** Got signal Terminated *** 2021-09-02T08:11:55.726 INFO:tasks.ceph.c1.osd.2.smithi175.stderr:2021-09-02T08:11:55.723+0000 7ff61ac88700 -1 osd.2 61 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-09-02T08:11:55.727 INFO:tasks.ceph.c1.mon.a.smithi175.stderr:2021-09-02T08:11:55.724+0000 7fafe8923700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster c1 -i a (PID: 30108) UID: 0 2021-09-02T08:11:55.727 INFO:tasks.ceph.c1.mon.a.smithi175.stderr:2021-09-02T08:11:55.724+0000 7fafe8923700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2021-09-02T08:11:55.728 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2021-09-02T08:11:55.724+0000 7f043df30700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 38676) UID: 0 2021-09-02T08:11:55.728 INFO:tasks.rgw.c1.client.1.smithi175.stdout:2021-09-02T08:11:55.724+0000 7f044ee6b440 -1 shutting down 2021-09-02T08:11:55.728 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2021-09-02T08:11:55.724+0000 7f71dd09c700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster c1 -i x (PID: 30111) UID: 0 2021-09-02T08:11:55.729 INFO:tasks.ceph.c1.mgr.x.smithi175.stderr:2021-09-02T08:11:55.724+0000 7f71dd09c700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2021-09-02T08:12:09.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ERROR 2021-09-02T08:12:09.976 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-54 2021-09-02T08:13:14.029 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ERROR 2021-09-02T08:13:14.030 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=efybpk-55 2021-09-02T08:13:39.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ERROR 2021-09-02T08:13:39.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:13:39.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T08:13:39.669 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T08:13:39.718 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T08:13:39.719 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T08:18:39.721 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T08:18:39.721+0000 7fba90c6b380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T08:23:39.721 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T08:23:39.722+0000 7fba90c6b380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T08:28:39.721 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T08:28:39.722+0000 7fba90c6b380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T08:33:39.721 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T08:33:39.723+0000 7fba90c6b380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T08:38:39.721 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T08:38:39.724+0000 7fba90c6b380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T08:43:39.720 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T08:43:39.724+0000 7fba90c6b380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T08:48:39.720 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T08:48:39.725+0000 7fba90c6b380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T08:53:39.721 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T08:53:39.726+0000 7fba90c6b380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T08:58:39.720 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T08:58:39.726+0000 7fba90c6b380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T09:03:39.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-02T09:03:39.726 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T09:03:39.726+0000 7fba90c6b380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T09:03:39.726 INFO:teuthology.orchestra.run.smithi175.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-02T09:03:39.728 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... FAIL 2021-09-02T09:03:39.729 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2021-09-02T09:03:39.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T09:03:39.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T09:03:39.740 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T09:03:39.790 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T09:03:39.791 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T09:08:39.792 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T09:08:39.799+0000 7f17e6bbe380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T09:13:39.793 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T09:13:39.799+0000 7f17e6bbe380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T09:18:39.792 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T09:18:39.800+0000 7f17e6bbe380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T09:23:39.793 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T09:23:39.800+0000 7f17e6bbe380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T09:28:39.794 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T09:28:39.801+0000 7f17e6bbe380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T09:33:39.793 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T09:33:39.801+0000 7f17e6bbe380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T09:38:39.792 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T09:38:39.801+0000 7f17e6bbe380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T09:43:39.797 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T09:43:39.801+0000 7f17e6bbe380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T09:48:39.798 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T09:48:39.804+0000 7f17e6bbe380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T09:53:39.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-02T09:53:39.810 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T09:53:39.803+0000 7f17e6bbe380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T09:53:39.811 INFO:teuthology.orchestra.run.smithi175.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-02T09:53:39.812 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... FAIL 2021-09-02T09:53:39.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T09:53:39.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T09:53:39.815 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T09:53:39.865 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T09:53:39.866 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T09:58:39.871 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T09:58:39.870+0000 7f5b4b6c1380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T10:03:39.873 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T10:03:39.870+0000 7f5b4b6c1380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T10:08:39.875 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T10:08:39.870+0000 7f5b4b6c1380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T10:13:39.877 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T10:13:39.870+0000 7f5b4b6c1380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T10:18:39.878 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T10:18:39.871+0000 7f5b4b6c1380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T10:23:39.878 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T10:23:39.872+0000 7f5b4b6c1380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T10:28:39.877 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T10:28:39.872+0000 7f5b4b6c1380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T10:33:39.877 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T10:33:39.872+0000 7f5b4b6c1380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T10:38:39.877 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T10:38:39.874+0000 7f5b4b6c1380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T10:43:39.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-02T10:43:39.888 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T10:43:39.874+0000 7f5b4b6c1380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T10:43:39.888 INFO:teuthology.orchestra.run.smithi175.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-02T10:43:39.890 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... FAIL 2021-09-02T10:43:39.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T10:43:39.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T10:43:39.894 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T10:43:39.944 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T10:43:39.944 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T10:48:39.947 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T10:48:39.943+0000 7eff5aa96380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T10:53:39.947 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T10:53:39.943+0000 7eff5aa96380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T10:58:39.948 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T10:58:39.944+0000 7eff5aa96380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T11:03:39.948 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T11:03:39.944+0000 7eff5aa96380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T11:08:39.949 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T11:08:39.945+0000 7eff5aa96380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T11:13:39.950 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T11:13:39.947+0000 7eff5aa96380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T11:18:39.950 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T11:18:39.947+0000 7eff5aa96380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T11:23:39.951 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T11:23:39.948+0000 7eff5aa96380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T11:28:39.952 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T11:28:39.948+0000 7eff5aa96380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T11:33:39.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-02T11:33:39.955 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T11:33:39.949+0000 7eff5aa96380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T11:33:39.955 INFO:teuthology.orchestra.run.smithi175.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-02T11:33:39.956 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... FAIL 2021-09-02T11:33:39.957 INFO:tasks.rgw_multisite_tests:test s3 notification filter on master ... ok 2021-09-02T11:33:39.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests_ps.test_object_timing ... ok 2021-09-02T11:33:39.958 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2021-09-02T11:33:39.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T11:33:39.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T11:33:39.961 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T11:33:40.010 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T11:33:40.011 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T11:38:40.012 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T11:38:40.010+0000 7fe3ffe9b380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T11:43:40.014 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T11:43:40.012+0000 7fe3ffe9b380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T11:48:40.015 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T11:48:40.013+0000 7fe3ffe9b380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T11:53:40.015 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T11:53:40.013+0000 7fe3ffe9b380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T11:58:40.015 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T11:58:40.013+0000 7fe3ffe9b380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T12:03:40.012 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T12:03:40.015+0000 7fe3ffe9b380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T12:08:40.004 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T12:08:40.015+0000 7fe3ffe9b380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T12:13:39.999 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T12:13:40.015+0000 7fe3ffe9b380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T12:18:39.995 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T12:18:40.015+0000 7fe3ffe9b380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T12:23:39.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-02T12:23:39.997 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T12:23:40.015+0000 7fe3ffe9b380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T12:23:39.997 INFO:teuthology.orchestra.run.smithi175.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-02T12:23:39.999 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... FAIL 2021-09-02T12:23:40.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T12:23:40.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T12:23:40.001 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T12:23:40.052 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T12:23:40.053 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T12:28:40.053 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T12:28:40.079+0000 7fcefafb6380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T12:33:40.051 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T12:33:40.079+0000 7fcefafb6380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T12:38:40.049 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T12:38:40.080+0000 7fcefafb6380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T12:43:40.049 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T12:43:40.080+0000 7fcefafb6380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T12:48:40.046 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T12:48:40.081+0000 7fcefafb6380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T12:53:40.045 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T12:53:40.082+0000 7fcefafb6380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T12:58:40.045 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T12:58:40.083+0000 7fcefafb6380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T13:03:40.043 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T13:03:40.083+0000 7fcefafb6380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T13:08:40.043 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T13:08:40.083+0000 7fcefafb6380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T13:13:40.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-02T13:13:40.061 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T13:13:40.083+0000 7fcefafb6380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T13:13:40.061 INFO:teuthology.orchestra.run.smithi175.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-02T13:13:40.062 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... FAIL 2021-09-02T13:13:40.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T13:13:40.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T13:13:40.065 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T13:13:40.115 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T13:13:40.115 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T13:18:40.133 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T13:18:40.144+0000 7fc1c685a380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T13:23:40.143 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T13:23:40.145+0000 7fc1c685a380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T13:28:40.150 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T13:28:40.146+0000 7fc1c685a380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T13:33:40.154 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T13:33:40.147+0000 7fc1c685a380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T13:38:40.158 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T13:38:40.147+0000 7fc1c685a380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T13:43:40.163 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T13:43:40.148+0000 7fc1c685a380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T13:48:40.159 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T13:48:40.149+0000 7fc1c685a380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T13:53:40.158 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T13:53:40.149+0000 7fc1c685a380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T13:58:40.158 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T13:58:40.150+0000 7fc1c685a380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T14:03:40.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-02T14:03:40.189 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T14:03:40.150+0000 7fc1c685a380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T14:03:40.189 INFO:teuthology.orchestra.run.smithi175.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-02T14:03:40.191 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... FAIL 2021-09-02T14:03:40.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T14:03:40.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T14:03:40.194 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T14:03:40.244 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T14:03:40.244 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T14:08:40.248 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T14:08:40.239+0000 7f468f3b7380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T14:13:40.249 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T14:13:40.240+0000 7f468f3b7380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T14:18:40.276 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T14:18:40.240+0000 7f468f3b7380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T14:23:40.248 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T14:23:40.241+0000 7f468f3b7380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T14:28:40.248 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T14:28:40.241+0000 7f468f3b7380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T14:33:40.247 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T14:33:40.242+0000 7f468f3b7380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T14:38:40.248 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T14:38:40.243+0000 7f468f3b7380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T14:43:40.249 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T14:43:40.243+0000 7f468f3b7380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T14:48:40.248 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T14:48:40.243+0000 7f468f3b7380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T14:53:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-02T14:53:40.254 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T14:53:40.243+0000 7f468f3b7380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T14:53:40.255 INFO:teuthology.orchestra.run.smithi175.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-02T14:53:40.256 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification on specific events ... FAIL 2021-09-02T14:53:40.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T14:53:40.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T14:53:40.259 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T14:53:40.308 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T14:53:40.308 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T14:58:40.309 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T14:58:40.305+0000 7fa6b9fbc380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T15:03:40.315 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T15:03:40.305+0000 7fa6b9fbc380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T15:08:40.308 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T15:08:40.305+0000 7fa6b9fbc380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T15:13:40.308 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T15:13:40.307+0000 7fa6b9fbc380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T15:18:40.309 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T15:18:40.306+0000 7fa6b9fbc380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T15:23:40.309 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T15:23:40.307+0000 7fa6b9fbc380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T15:28:40.309 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T15:28:40.307+0000 7fa6b9fbc380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T15:33:40.309 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T15:33:40.307+0000 7fa6b9fbc380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T15:38:40.309 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T15:38:40.308+0000 7fa6b9fbc380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T15:43:40.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-02T15:43:40.313 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T15:43:40.309+0000 7fa6b9fbc380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T15:43:40.313 INFO:teuthology.orchestra.run.smithi175.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-02T15:43:40.315 INFO:tasks.rgw_multisite_tests:test set/get/delete of subscription ... FAIL 2021-09-02T15:43:40.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T15:43:40.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T15:43:40.318 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T15:43:40.367 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T15:43:40.367 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T15:48:40.370 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T15:48:40.370+0000 7f67e64bf380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T15:53:40.370 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T15:53:40.370+0000 7f67e64bf380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T15:58:40.370 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T15:58:40.371+0000 7f67e64bf380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T16:03:40.372 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T16:03:40.371+0000 7f67e64bf380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T16:08:40.375 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T16:08:40.372+0000 7f67e64bf380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T16:13:40.375 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T16:13:40.374+0000 7f67e64bf380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T16:18:40.375 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T16:18:40.374+0000 7f67e64bf380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T16:23:40.376 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T16:23:40.374+0000 7f67e64bf380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T16:28:40.377 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T16:28:40.375+0000 7f67e64bf380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T16:33:40.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-02T16:33:40.381 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T16:33:40.376+0000 7f67e64bf380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T16:33:40.382 INFO:teuthology.orchestra.run.smithi175.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-02T16:33:40.383 INFO:tasks.rgw_multisite_tests:test radosgw-admin commands ... FAIL 2021-09-02T16:33:40.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T16:33:40.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T16:33:40.386 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T16:33:40.437 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T16:33:40.437 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T16:38:40.438 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T16:38:40.437+0000 7fd08af06380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T16:43:40.437 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T16:43:40.437+0000 7fd08af06380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T16:48:40.436 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T16:48:40.438+0000 7fd08af06380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T16:53:40.436 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T16:53:40.438+0000 7fd08af06380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T16:58:40.436 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T16:58:40.439+0000 7fd08af06380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T17:03:40.436 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T17:03:40.439+0000 7fd08af06380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T17:08:40.436 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T17:08:40.440+0000 7fd08af06380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T17:13:40.438 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T17:13:40.440+0000 7fd08af06380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T17:18:40.441 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T17:18:40.441+0000 7fd08af06380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T17:23:40.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-02T17:23:40.450 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T17:23:40.442+0000 7fd08af06380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T17:23:40.451 INFO:teuthology.orchestra.run.smithi175.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-02T17:23:40.452 INFO:tasks.rgw_multisite_tests:test that events are only sent on incremental sync ... FAIL 2021-09-02T17:23:40.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T17:23:40.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T17:23:40.455 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T17:23:40.504 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T17:23:40.504 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T17:28:40.508 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T17:28:40.506+0000 7f24cf584380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T17:33:40.508 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T17:33:40.506+0000 7f24cf584380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T17:38:40.509 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T17:38:40.507+0000 7f24cf584380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T17:43:40.511 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T17:43:40.507+0000 7f24cf584380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T17:48:40.511 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T17:48:40.508+0000 7f24cf584380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T17:53:40.512 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T17:53:40.508+0000 7f24cf584380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T17:58:40.513 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T17:58:40.510+0000 7f24cf584380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T18:03:40.514 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T18:03:40.511+0000 7f24cf584380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T18:08:40.513 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T18:08:40.512+0000 7f24cf584380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T18:13:40.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-02T18:13:40.518 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T18:13:40.512+0000 7f24cf584380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T18:13:40.518 INFO:teuthology.orchestra.run.smithi175.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-02T18:13:40.520 INFO:tasks.rgw_multisite_tests:test subscriptions for different events ... FAIL 2021-09-02T18:13:40.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T18:13:40.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T18:13:40.523 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T18:13:40.575 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T18:13:40.576 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T18:18:40.579 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T18:18:40.577+0000 7f816f527380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T18:23:40.580 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T18:23:40.577+0000 7f816f527380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T18:28:40.580 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T18:28:40.577+0000 7f816f527380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T18:33:40.580 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T18:33:40.579+0000 7f816f527380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T18:38:40.581 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T18:38:40.579+0000 7f816f527380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T18:43:40.582 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T18:43:40.579+0000 7f816f527380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T18:48:40.582 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T18:48:40.579+0000 7f816f527380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T18:53:40.582 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T18:53:40.580+0000 7f816f527380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T18:58:40.583 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T18:58:40.581+0000 7f816f527380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T19:03:40.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-02T19:03:40.588 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T19:03:40.581+0000 7f816f527380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T19:03:40.588 INFO:teuthology.orchestra.run.smithi175.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-02T19:03:40.590 INFO:tasks.rgw_multisite_tests:test incremental fetching of events from a subscription ... FAIL 2021-09-02T19:03:40.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T19:03:40.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-02T19:03:40.593 DEBUG:teuthology.orchestra.run.smithi175:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-02T19:03:40.641 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setuser ceph since I am not root 2021-09-02T19:03:40.642 INFO:teuthology.orchestra.run.smithi175.stderr:ignoring --setgroup ceph since I am not root 2021-09-02T19:08:40.645 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T19:08:40.642+0000 7fbd4894f380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T19:13:40.645 INFO:teuthology.orchestra.run.smithi175.stderr:2021-09-02T19:13:40.643+0000 7fbd4894f380 0 monclient(hunting): authenticate timed out after 300 2021-09-02T19:17:10.520 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-09-02T19:17:10.547 DEBUG:teuthology.task.console_log:Killing console logger for smithi175 2021-09-02T19:17:10.548 DEBUG:teuthology.task.console_log:Killing console logger for smithi197 2021-09-02T19:17:10.548 DEBUG:teuthology.exit:Finished running handlers